Search code examples
shellhadoopclouderaoozieoozie-coordinator

Move file from HDFS one directory to other directory in HDFS using OOZIE?


I am trying to copy a file from HDFS one directory to other directory in HDFS, with the help of shell script as a part of oozie Job, but i am not able to copy it through oozie.

Can we copy file from HDFS one directory to other director in HDFS using oozie.

when i am running the oozie job, i am not any getting error.

it is showing status SUCCEEDED but file is not copying to destination directory.

oozie Files are below.

test.sh

#!/bin/bash
echo "listing files in the current directory, $PWD"
sudo hadoop fs -cp /user/cloudera/RAVIOOZIE/input/*  /user/cloudera/RAVIOOZIE/output/
ls  # list files

my workflow.xml is

<workflow-app name="RAMA" xmlns="uri:oozie:workflow:0.5">
<start to="shell-381c"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-381c">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>/user/cloudera/test.sh</exec>
        <file>/user/cloudera/test.sh#test.sh</file>
          <capture-output/>
    </shell>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

and my job.properties

oozie.use.system.libpath=True
security_enabled=False
dryrun=True
jobTracker=localhost:8032
nameNode=hdfs://quickstart.cloudera:8020
oozie.wf.application.path=${nameNode}/user/cloudera/test/

please help on this. why file is not copying to my destination director.

please let me know is there any thing i missed.


Solution

  • As mentioned in the comments by @Samson:

    If you want to do hadoop actions with oozie, you should use a hdfs action rather than a shell action for that.


    I am not sure why you don't get an error, but here is some speculation on what might happen:

    You give oozie the task of starting a shell action, it succesfully starts the shell action and reports a success. Then the shell action fails, but that's not oozies problem.