Search code examples
pythonpython-3.xairflowcentos7rc

Run apache airflow worker after boot

I tried to setup "airflow worker" to run after system start via rc.local(centos 7).

I have installed python and airflow as root. Path is /root/airflow and /root/anaconda3.

I added this to rc.local:

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
exec 2> /home/centos/rc.local.log  # send stderr from rc.local to a log file
exec 1>&2                      # send stdout to the same log file
set -x                         # tell sh to display commands before execution
export C_FORCE_ROOT="true"
/root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
exit 0

When I try it to run manually it works (sh /etc/rc.local)

But when it runs after boot, it crashes with this error in log file.

It seems like it can't find path to airflow, but I have written it in full.

+ export C_FORCE_ROOT=true
+ C_FORCE_ROOT=true
+ /root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
Traceback (most recent call last):
  File "/root/anaconda3/bin/airflow", line 37, in <module>
    args.func(args)
  File "/root/anaconda3/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1129, in worker
    sp = _serve_logs(env, skip_serve_logs)
  File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1065, in _serve_logs
    sub_proc = subprocess.Popen(['airflow', 'serve_logs'], env=env, close_fds=True)
  File "/root/anaconda3/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/root/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'airflow': 'airflow'
Solution

  • A place to start is to change this line...

    /root/anaconda3/bin/python /root/anaconda3/bin/airflow worker

    to

    /root/anaconda3/bin/airflow worker

    As you only need to invoke the airflow bin you need and pass it a single service. Bear in mind you can pass more arguments. But calling a version of Python doesn't feel necessary.