Search code examples
phplinuxprocessbackground

How to run a longlife background process, linux server


I am running a webapp (php) that needs to process some "heavy-work" tasks (from 30 sec to 3 min). I know it is not extremely heavy, but I cannot have my user waiting for them, so I've set up an internal API like: http://localhost/process-picture/745884/ and I store this operation in a MySQL table.

Now I want to create a "process" that fetches that MySQL table and performs the oldest queued operation, and once it is done, it get the next, and so on.

First of all I thought about making a PHP Script that calls by cURL the system like that:

fetchOperation.php connects to DB and gets the URL to the operation to call it by cURL. Every operation: performs itself, and after that deletes itself from the queue and calls (cURL) fetchOperation.php again.

I feel that this system is kinda tricky, so I was wondering if there is (and in which language to write it) any way to set up a background process that checks the DB every 15 sec, and does the following:

  • Delete all rows marked as DONE.
  • Check if there is any row marked as PROCESSING, if so, just exit and wait for the next 15 sec.
  • If there's no PROCESSING row, fire the oldest PENDING one (FIFO queue).

This way I can manage what is being processed at any time, and even control the server load (for instance, at night allow to have up to three PROCESSING items).

Sorry for such a long explanation, and thanks in advance!


Solution

  • Finally I solved it that way:

    1st. Design the operations in a way that they are url callable. For instance: http://localhost/render-image/14523665

    2nd. Store pending operations in a table like this: operations(opertation-id, operation-url, operation-status).

    Once this was ready I designed a short PHP script that does the following:

    <?php
    if( WORKER_IS_ONLINE !== true ) exit();
    OperationsQueue::CleanQueue();
    while( OperationsQueue::RunningOperations() < WORKER_MAX_OPERATIONS ) {
        launch_operation( OperationsQueue::DequeOperation() );
    }
    sleep( WORKER_SLEEP_TIME );
    launch_operation( '/operations_queue/deque' );
    

    This auxiliar functions (launch_operation and the OperationsQueue class) do as follows (notice it's not yet implemented):

    <?php 
    define('WORKER_SLEEP_TIME', 15);     // Time to sleep between sweeps.
    define('WORKER_HOST', 'localhost');  // Hostname where the operations will be performed.
    define('WORKER_PORT', 80);       // Port where the operations will be performed.
    define('WORKER_SOCKET_TIMEOUT', 80); // Port where the operations will be performed.
    define('WORKER_MAX_OPERATIONS', 2);  // Max simultaneous operations.
    define('WORKER_MAX_LAUNCHES', 2);    // Max operations launched within a sweep.
    define('WORKER_IS_ONLINE', false);   // Bool value that marks whether the worker must be working or not.
    
    function launch_operation($operation) {
        $fp = fsockopen(WORKER_HOST, WORKER_PORT, $errno, $errstr, WORKER_SOCKET_TIMEOUT);
        if ($fp) {
            $out  = 'GET ' . $operation . " HTTP/1.1\r\n";
            $out .= 'Host: ' . WORKER_HOST . "\r\n\r\n";
            fwrite($fp, $out);
            fclose($fp);
        }
    }
    
    class OperationsQueue {
    
        public static function RunningOperations(){
            // connect to DB an perform: SELECT COUNT(*) FROM operations WHERE status = 'PROCESSING';
            return 1;
        }
        public static function CleanQueue(){
            // connect to DB an perform: DELETE FROM operations WHERE status IN ('DONE', 'CANCELED');
        }
        public static function DequeOperation(){
            // connect to DB an perform: SELECT * FROM operations WHERE status = 'PENDING' ORDER BY id ASC LIMIT 0, 1;
            // UPDATE operations SET status = 'PROCESSING', tries = tries+1 WHERE id = $id;
            return 'operation_url';
        }
    
    }
    

    I thought it could be useful to some other people. As you see it chains the operations.