sidekiqctl stop halting the further execution of commands in preStop on devtron

Question

Trying to run sidekiqctl stop in preStop hook of pod lifecycle, doing so to gracefully shutdown sidekiq process while pod termination.

Script that I'm trying to execute in preStop hook

    echo "Stopping sidekiq..."
    for pid in $SIDEKIQ_PIDS/*.pid; do
      echo "Stopping specific pid..."
      bundle exec sidekiqctl stop $pid 30
      echo "removing pid file"
      rm -rf $pid
    done
    echo "done"

Expected behaviour:

Sidekiq process should gracefully shutdown and all the echo statement should be printed

Actual behaviour:

it's running fine till sidekiqctl stop command. I have the checked the pod logs, sidekiq shutdown logs are getting captured but no further echo command are getting executing that leads to preStopFailed event.

Pod logs:

Note:

Have tried various method to debug the sidekiqctl stop commands output like running it in background then waiting for that that process to compelete and Also tried by increasing the graceTimeout and sidekiqctl command timeout still the same issue.

Answer 1

After a lot of hit and trial, reached on below conclusion

• Once the main process of the container gets shutdown, k8 does not wait for completion of preStop hook script, it will instantly move to next state and kill the container and pod further.
• In my case once sidekiq process got shutdown, before sidekiqctl command completion pod got terminated, that cause the preStopHook failed
• To solve the issue, added & in end of sidekiqctl command to run it as a background process, now preStop hook will also complete successfully and graceful shutdown will also be & triggered in parallel, incase any issue in sidekiq shutdown, GracePeriod will take care of it by forcefully killing it.

NOTE: don't forget to add >> /proc/1/fd/1 2>&1 for log redirection, it helped in debugging.