How to handle SIGTERM signal on laravel 8,9,10 deploy with docker containers in regards to scheduler and worker?

Question

I saw that Laravel 8,9,10 handles the queue worker on SIGTERM signal. But according to the bellow comment from Worker class (Worker.php on line 185), laravel handles it only for the current worker execution, if supervisor (or other monitoring tool used) is not altered to take into account that SIGTERM and to NOT start the worker again, then it has little effect because the worker is stopped from laravel after he finishes the execution.(also artisan queue:restart can be issued when the supervisor stops restarting the queue, so that all jobs are gracefully stopped).

        // Finally, we will check to see if we have exceeded our memory limits or if
        // the queue should restart based on other indications. If so, we'll stop
        // this worker and let whatever is "monitoring" it restart the process.

How about the scheduler? How can I prevent (from within Laravel) the \App\Console\Kernel::schedule function to start the commands if a SIGTERM signal has been received?

Basically

php artisan schedule:run

should do nothing after SIGTERM signal.

When the docker container will be shut down, it receives this signal. I want to stop any new scheduled comands on that container from starting. This happens on a deploy or on a scale down scenario. I don't want the container to be killed in the middle of a process.

Acc. to https://stackoverflow.com/a/53733389/7309871 (if nothing changed since 2018) running the comands in a queue would solve this problem but restricts the scheduler to jobs only.

Since Laravel uses symfony https://symfony.com/blog/new-in-symfony-5-2-console-signals

If you prefer to handle some signals for all application commands (e.g. to log or profile commands), define an event listener or subscriber and listen to the new ConsoleEvents::SIGNAL event.

But this can't handle it anyway because:

This could be a possible solution for the kernel but as its window of execution if very narrow and because this is executed each minute missing the SIGTERM sent at second 30 for example, a better way would be to stop execution of php artisan schedule:run server side after SIGTERM signal is received.

\App\Console\Kernel.php

/**
 * @inheritdoc
 */
public function __construct(Application $app, Dispatcher $events)
{
    parent::__construct($app, $events);
    $this->app->singleton(SignalSingleton::class);

    if (!\extension_loaded('pcntl')) {
        return;
    }

    \pcntl_async_signals(true);
    \pcntl_signal(SIGTERM, function (int $signo, mixed $siginfo): void {
        Log::info('SIGTERM received');
        \resolve(SignalSingleton::class)->shedulerIsEnabled = false;
    });
}

/**
 * Define the application's command schedule.
 */
protected function schedule(Schedule $schedule): void
{
    if (!\resolve(SignalSingleton::class)->shedulerIsEnabled) {
        return;
    }
...

The SignalSingleton class should contain only a public bool $shedulerIsEnabled = true;

Answer 1

Most people run their queue workers in supervisord setups, which has some downfalls like not supporting delayed startups (unless you add a script w/ a sleep), and is unable to handle FATAL crashes, which means the queue worker will crash and supervisor won't restart it (ever seen your queues fill up? :)

There's a solution I just started using, running the below script as the CMD of a php-fpm docker-container.

benefits:

• no need to add supervisord to the php-fpm container (saves lots of resources)
• queues will for sure be restarted on failure
• can handle graceful shutdown during auto-scaling events by trapping signals
• it's simple and efficient

#!/usr/bin/env bash

work=true

function stopQueues() {
    echo "Stopping queues..."
    work=false
    php artisan queue:restart
    sleep 300
    exit 0
}

# handle graceful shutdown
trap "" SIGPIPE
trap stopQueues SIGTERM SIGINT SIGHUP

echo "Starting PHP queue workers..."
# SQS_QUEUE (bg)
(while $work; do
    php artisan queue:work --queue="$(grep -oP 'SQS_QUEUE=\K.*' .env)" --no-interaction
    sleep 1
done &)

# SQS_FALLBACK_QUEUE (bg)
(while $work; do
    php artisan queue:work --queue="$(grep -oP 'SQS_FALLBACK_QUEUE=\K.*' .env)" --no-interaction
    sleep 1
done &)

# SQS_NOTIFICATION_QUEUE (fg) blocking
while $work; do
    php artisan queue:work --queue="$(grep -oP 'SQS_NOTIFICATION_QUEUE=\K.*' .env)" --no-interaction
    sleep 1
done