Search code examples
hadoopsolrhdfsapache-zookeeperoozie

Sending solr configs to zookeeper from hdfs using zkcli.sh bash through oozie shell action


I get exception below, every time i try to send solr configs to zookeeper.

As confdir I'm putting this address: "hdfs://some_address/user/Peter/dir_with_date".

In implementation in jars behind bash script i see that there is a Paths.get method, which is transforming my string with configs to one without double slashes ("hdfs:/some_ad...).

Exception in thread "main" java.io.IOException: Path hdfs:/some_address/user/Peter/dir_with_date does not exist
  at org.apache.solr.common.cloud.ZkConfigManager.uploadToZK(ZkConfigManager.java:56)
  at org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:120)
  at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:220)

Am I doing something wrong or is these zkcli not suitable to read data from hdfs? What's the best way to deal with such situation (building a custom jar) ?


Solution

  • Thank you @Manjunath, that helped. Here is exact solution:

    To send solr configs to zookeeper through oozie shell action you have to copy solr config dir to worker executing this bash script by adding this into workflow

     <file>yourpathto/solr_config_dir#$yourpathto/solr_config_dir</file>
    

    And than in bashscript you refer to this folder like this:

    CONFIGDIR="`readlink -f solr_config_dir`"
    

    Add CONFIGDIR as confdir parameter to zkcli.sh and done!

    Now you have working oozie shell action that will send your solr configs to zookeeper!