Search code examples
shellhdfsapache-pigoozie

Checking if a file exists in HDFS location with size greater than zero, using Oozie

I have a Oozie workflow containing a Pig action generating a part-file as output as

/user/wf_user/app_dir/output/part-v003-o000-r-00100

After the Pig action, there is an fs-action which is generating a completion-flag file and moving part-v003-o000-r-00100 to alert_message (for renaming) and then change access permission on the file path /user/wf_user/app_dir/output/alert_message to make the file accessible from subsequent workflow actions.

Thereafter, there is a decision control node to check if the file /user/wf_user/app_dir/output/alert_message exists and is having size greater than zero. Only if the size is non-zero, the alert message will be emailed.

But even though the file exists and is having non-zero size, the decision condition is always returning false, thereby the alert message is never emailed to the notify users.

<switch xmlns="uri:oozie:workflow:0.4">
  <case to="message_pref_alert">false</case>
  <default to="success_email" />
</switch>

Below is the snippet of the relevant workflow actions

    <action name='generate_preftable_pref_count_report' cred="hcatauth,athensauth">
        <pig>
        <prepare>
        <delete path="${flag_dir}"></delete>
        </prepare>
            <script>generate_diffcount_w_perc_mktg_prefs.pig</script>
            <param>today=${today}</param>
            <param>prev_date=${prev_date}</param>
            <param>lake_tahoe_dump=${lake_tahoe_dump}</param>
            <param>current_pref_snapshot=${current_pref_snapshot}</param>
            <param>preference_user=${preference_user}</param>
            <param>pref_count_report=${pref_count_report}</param>
            <param>flag_dir=${flag_dir}</param>
            <file>${common_lib}/elephant-bird-pig.jar#elephant-bird-pig.jar</file>
            <file>${common_lib}/elephant-bird-core.jar#elephant-bird-core.jar</file>
            <file>${common_lib}/elephant-bird-hadoop-compat.jar#elephant-bird-hadoop-compat.jar</file>
        </pig>
        <ok to="fs-create-report-flag-and-alert" />
        <error to="failure_email" />
    </action>

    <action name="fs-create-report-flag-and-alert">
    <fs>
       <chmod path='${flag_dir}' permissions='-rwxrwxrwx' dir-files='true'/>
       <delete path='${flag_dir}/report_generated_${prev_date}'/>
       <touchz path='${flag_dir}/report_generated_${today}'/>
       <move source='${flag_dir}/part*' target='${alert_message_file}' />
       <chmod path='${alert_message_file}' permissions='-rwxrwxrwx' dir-files='true'/> 
    </fs>
    <ok to="if_alert_prefs_present"/>
    <error to="failure_email"/>
    </action>

    <decision name="if_alert_prefs_present">
        <switch>
    <case to="message_pref_alert">${(fs:exists('${alert_message_file}')) and (fs:fileSize('${alert_message_file}') gt 0 )}</case>
    <default to="success_email"/>
    </switch>
    </decision>

    <action name="message_pref_alert">
                <email xmlns="uri:oozie:email-action:0.1">
                        <to>${notify_to}</to>
                        <subject>PrefTable-PrefCount-Update-${today} : Pref Count Alert</subject>
                        <body>In today's pref counts, the pref count difference >= 5% for some sub/unsub preferences. Please check the below file for sub/unsub details.
                  ${alert_message_file} 
                  For further details on count and percentage difference, please check the hive table ${pref_count_report} .
            </body>
                </email>
                <ok to="success_email"/>
                <error to="failure_email"/>
     </action>

Note: I have not set the ${alert_message_file} as an workflow configuration property but have set it only in the coordinator properties file as below.

nameNode=hdfs://clusterName-nn1.clusterDomain.com:8020
jobTracker=clusterName-jt1.clusterDomain.com:8032
MAIN_DIR=/user/wf_user
APP_DIR=${nameNode}${MAIN_DIR}/app_dir
flag_dir=${APP_DIR}/output
alert_message_file=${flag_dir}/alert_message

Looked into other similar discussions on SO regarding the same topic: How to check whether the file exist in HDFS location, using oozie?

Solution

  • Passed the ${alert_message_file} as a workflow configuration property to get the error resolved. Actually the variable alert_message_file (defined in properties) was not accessible unless it is passed through configuration property through workflow, else there would be "variable could not be resolved" error.

    In workflow.xml

    <configuration>
    <property>
        ...
    </property>  
    <property>
        <name>alert_message_file</name>
        <value>${alert_message_file}</value>
    </property>
</configuration>

    Then changed the decision node as follows

    <decision name="if_alert_prefs_present">
    <switch>
<case to="message_pref_alert">${(fs:exists(wf:conf('alert_message_file'))) and (fs:fileSize(wf:conf('alert_message_file')) gt 0 )}</case>
<default to="success_email"/>
</switch>
</decision>