Search code examples
pythonhdfsoozie

How do I import a local python module when running a python script in Oozie?


I have two python files - my_python_A.py and my_python_B.py. The first file references the second (from my_python_B import *).

I'm executing the first python file from a shell action in Oozie (i.e. the script is simply python my_python_A.py), and am receiving the following error:

Traceback (most recent call last):
  File "my_python_A.py", line 2, in <module>
    from my_python_B import *
ImportError: No module named my_python_B
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Both python files are located under the same directory in HDFS. How can I get this import statement to work?


Solution

  • I faced the same issue and the way I worked around this problem was by setting the environment variable PYTHONPATH to the current working directory inside the shell script before I execute my python code

    export PYTHONPATH=`pwd`
    python m_python_A.py
    

    Make sure that in your shell action you have included all the required python modules inside the <file></file> tags. Assuming that you have a shell script called sample_script.sh (inside which you have the aforementioned commands) your workflow.xml file should look something like this

    <workflow-app name="shellTest" xmlns="uri:oozie:workflow:0.4">
        <start to="shell-action"/>
        <action name="shell-action">
            <shell xmlns="uri:oozie:shell-action:0.2">
                <job-tracker>${jobTracker}</job-tracker>
                <name-node>${nameNode}</name-node>
                <configuration>                
                    <property>
                        <name>oozie.launcher.mapred.job.queue.name</name>
                        <value>${launcherqueue}</value>
                    </property>
                    <property>
                        <name>mapred.job.queue.name</name>
                        <value>${mapredqueue}</value>
                    </property>
                </configuration>
                <exec>sample_script.sh</exec>
                <file>${appPath}/sample_script.sh#sample_script.sh</file>
                <file>${appPath}/m_python_A.py#m_python_A.py</file>
                <file>${appPath}/m_python_B.py#m_python_B.py</file>
                <capture-output/>
            </shell>
            <ok to="end"/>
            <error to="shell-action-failed"/>
        </action>
    
        <kill name="shell-action-failed">
            <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
    
        <end name="end" />
    
    </workflow-app>