Search code examples
phpbatch-fileresque

Start worker job from where it was left off with php resque?


Im batching queries in a php job to process. I am curious if there is a way to pick up where the job left off at.

psdeuo code

function perform(){
    $batch_limit = 0;
    while($batch_limit<20){
       results = getstuff($batch_limit);
       dostuff(results);
       $batch_limit += 2;
}    

function getstuff($batch_limit){
   $results = $con->query("SELECT * FROM table LIMIT $batch_limit, 2);
   return $results;
}

So for example if the arbitrary dostuff() function breaks on a result query of

'Select * FROM table LIMIT 4, 2'

Is there anyway i can pick up and execute

'Select * From table LIMIT 6, 2'


Solution

  • Write the current limit to a simple key/value table and SELECT it at the beginning of perform().

    I always have a table for this kind of information, because it often comes in handy. Just create a table with two columns "key", and "value". Key being VARCHAR and value being integer.

    Then you insert your batch limit at the beginning of each call to getstuff()

    function getstuff($batch_limit){
        $con->query("UPDATE keyvaluetable SET value=$batch_limit WHERE key='batch_limit'");
        $results = $con->query("SELECT * FROM table LIMIT $batch_limit, 2");
        return $results;
    }
    

    And select it at the beginning of your perform()

    function perform(){
        $result = $conn->query("Get value from before query here");
        if($result){
           $batch_limit = $result;
        }else{
           $batch_limit = 0;
        }
        while($batch_limit<20){
           results = getstuff($batch_limit);
           dostuff(results);
           $batch_limit += 2;
    }