Search code examples
hadoophiveactivemq-classicazure-hdinsighthcatalog

HCatalog notifications on HDInsight

I am trying to enable HCatalog notifications on an HDInsight cluster so that I can keep track of the metadata changes.

I have an ActiveMQ broker running on both of my headnodes. It is working as I have been able to subscribe to it and receive notifications that I was sending to myself. I tried enabling the HCatalog notifications by changing the configuration of the hive-site.xml through the Ambari UI (I added org.apache.hive.hcatalog.listener.NotificationListener in the hive.metastore.event.listeners field) but that didn't give any results. I also connected to the headnode of my HDInsight cluster via SSH and manually changed the configuration in hive-site.xml files (there are apparently quite a few of these) by adding a property like I tried through Ambari and event that didn't work. After every configuration change I restarted hive (and even all) services through the Ambari UI. My guess is that, for some reason, the changes that I made are not being applied.

Any experience or advice on this?

Solution

  • I managed to activate a custom notification listener by doing the following:

    1. Package the class which implements the MetaStoreEventListener interface into a jar
    2. Place the jar into the lib folder of hive-server on every headnode

    3. In the Ambari UI, go to Hive => Configs => Advanced => Custom hive-site and add the property:

      hive.metastore.event.listeners=[full_class_name_including_package]

    4. Save the configuration (if warnings occur, click Proceed Anyway)

    5. Restart all affected services when prompted.

    After going through these steps the metastore called my notification listener after every DDL operation and I was able to keep track of all the metadata changes.