Search code examples
hadoopjenkinsjenkins-pluginsoozie

Deploy Oozie jobs via Jenkins

I am writing Oozie jobs and I want to deploy them to a hadoop cluster.

However rather than doing them manually, I want to use Jenkins to do this deployment for me.

I searched the web and didn't find any plugin for oozie .... so if you are using oozie on a hadoop cluster, can you please tell me what is the best way in which I could do automated build & deployments using Jenkins?

Solution

  • Deployment of Oozie workflows is just putting xml files and required dependencies not available on the cluster to HDFS directory. link

    • Assuming that you have two Jenkins jobs,
      • one for build, test etc
      • second for deployment which is triggered after sucesful build of first job

    you can script putting files on HDFS in this second job for example using WebHdfs

    Other way if product of your build from first job is some kind of a package e.g. zip, you can create a script which will download this artifact from Jenkins/Nexus to machine with access to hdfs console command, extract this zip and just put whole directory to hdfs, as its described here

    If you are using some provisioning tool (Puppet or Ansible) you can use it for this purpose.

    For configuring Jenkins scheduling you may want to use

    • Jenkins Pipeline Plugin
    • or on Jenkins Job page "Configure -> Build Triggers -> Build after other projects are built"