Why would hot deploy of Hypnotoad rerun old http requests?

Question

The nutshell:
When I do a hot deployment of Hypnotoad sometimes the new server immediately processes a slew of HTTP requests that were already handled by the previous server.

If a response has been rendered but the thread is still doing some processing does Mojo/Hypnotoad retain the request until the processing has stopped? Do I need to tell the server that the HTTP request is resolved?

The long version:
I have a Mojolicious::Lite app running under Hypnotoad. The app's function is to accept HTTP requests from another service.

We are processing jobs that progress through a series of states. At each job state change the app is notified with an HTTP request. This is a busy little script - recieving more than 1000 req/hour.

The scripts job is to manipulate some data .. doing DB updates, editng files, sending mail.

In an effort to keep things moving along, when it recieves the HTTP request it sanity checks the data it recieved. If the data looks good it sends a 200 response to the caller immediately and then continues on to do the more time consuming tasks. (I'm guessing this is the underlying cause)

When I hot deploy - by rerunning the start script (which runs 'localperl/bin/hypnotoad $RELDIR/etc/bki/bki.pl') - some requests that were already handled are sent to the new server and reprocessed.

Why are these old transactions still being held by the original server? Many have been long since completed! Does the need to tell Mojolicious that the request is done before it goes off and messes with data? (I considered $c->finish() but that is just for sockets?) How does Hypnotoad decide what requests should be passed to it's replacement server?

Here is some psuedo code with what I'm doing:

get '/jobStateChange/:jobId/:jobState/:jobCause' => sub {
    my $c =shift;

    my $jobId = $c->stash("jobId");
    return $c->render(text => "invalid jobId: $jobId", status => 400) unless $jobId=~/^\d+$/;

    my $jobState = $c->stash("jobState");
    return $c->render(text => "invalid jobState: $jobState", status => 400) unless $jobState=~/^\d+$/;

    my $jobCause = $c->stash("jobCause");
    return $c->render(text => "invalid jobCause: $jobCause", status => 400) unless $jobCause=~/^\d+$/;

    my $jobLocation = $c->req->param('jobLocation');
    if ($jobLocation){ $jobLocation = $ENV{'DATADIR'} . "/jobs/" . $jobLocation; }
    unless ( $jobLocation && -d $jobLocation ){
        app->log->debug("determining jobLocation because passed job jobLocation isn't useable");
        $jobLocation = getJobLocation($jobId);
        $c->stash("jobLocation", $jobLocation);
    }

    # TODO - more validation? would BKI lie to us?
    return if $c->tx->res->code && 400 == $c->tx->res->code; # return if we rendered an error above

    # tell BKI we're all set ASAP
    $c->render(text => 'ok');
    handleJobStatusUpdate($c, $jobId, $jobState, $jobCause, $jobLocation);

};

sub handleJobStatusUpdate{
    my ($c, $jobId, $jobState, $jobCause, $jobLocation) = @_;
    app->log->info("job $jobId, state $jobState, cause $jobCause, loc $jobLocation");

    # set the job states in jobs
    app->work_db->do($sql, undef, @params);


    if ($jobState == $SOME_JOB_STATE) { 
        ... do stuff ...
        ... uses $c->stash to hold data used by other functions
    }

    if ($jobState == $OTHER_JOB_STATE) { 
        ... do stuff ...
        ... uses $c->stash to hold data used by other functions
    }
}

Answer 1

Your request will not be complete until the request handler returns. This little app, for example, will take 5 seconds to output "test":

# test.pl
use Mojolicious::Lite;
get '/test' => sub { $_[0]->render( text => "test" ); sleep 5 };
app->start;

The workaround for your app would be to run handleJobStatusUpdate in a background process.

get '/jobStateChange/:jobId/:jobState/:jobCause' => sub {
    my $c =shift;    
    my $jobId = $c->stash("jobId");
    my $jobState = $c->stash("jobState");
    my $jobCause = $c->stash("jobCause");
    my $jobLocation = $c->req->param('jobLocation');
    ...
    $c->render(text => 'ok');
    if (fork() == 0) {
        handleJobStatusUpdate($c, $jobId, $jobState, $jobCause, $jobLocation);
        exit;
    }