Search code examples
kubernetescelerysh

celery worker in kubernetes won't teminate while running in sh -c

I have an issue running celery worker using shin kuberntes. How do I make the celery worker receive TERMSIG?

kubernetes Celery worker doesn't receive TERMSIG while running under sh -c manifest example:

...
command: [ 'sh']
args: [ '-c', 'celery --quiet -A ${CELERY_APP} worker --prefetch-multiplier 1 --loglevel ${WORKER_LOGLEVEL} --queues ${WORKER_QUEUE} -n ${WORKER_PREFIX}-${HOSTNAME} -c 1 2>&1' ]
...

When the pod terminates the worker doesn't indicate it received TERMSIG

Connecting to the pods and running kill terminates the worker normally.

if I run the worker like:

...
command: ["celery"]
args: ["--quiet", "-A", "$(CELERY_APP)", "worker" , "--prefetch-multiplier", "1", "--loglevel", "$(WORKER_LOGLEVEL)", "--queues", "$(WORKER_QUEUE)", "-n", "$(WORKER_PREFIX)-$(HOSTNAME)"]
...

The shutdown works fine (Warm shutdown)

Solution

  • Sh doesn't propagate signals to children. I've end up using a bash script like specified at https://veithen.io/2014/11/16/sigterm-propagation.html

    propagating the signals manually

    #!/bin/bash
trap 'kill -TERM $PID; wait $PID' TERM INT

celery --quiet -A ${CELERY_APP} worker --prefetch-multiplier 1 --loglevel ${WORKER_LOGLEVEL} --queues ${WORKER_QUEUE} -n ${WORKER_PREFIX}-${HOSTNAME} -c 1 2>&1 &
PID=$!
wait $PID
wait $PID
EXIT_STATUS=$?