Fetching data simultaneously in Mojolicious

Question

I'm trying to run multiple subroutines in Parallel (to fetch data from external systems). To simulate, I use sleep in my example below. My question is: how can I achieve this in Mojolicious?

#!/usr/bin/perl
use Mojolicious::Lite;
use Benchmark qw(:hireswallclock);

sub  add1 { my $a = shift; sleep 1; return $a+1; }
sub mult2 { my $b = shift; sleep 1; return $b*2; }
sub power { my ($x, $y) = @_; sleep 1; return $x ** $y; }

any '/' => sub {    
    my ( $self ) = @_;

    my $n = int(rand(5));

    my $t0 = Benchmark->new;
    my $x = mult2($n); # Need to run in parallel
    my $y =  add1($n); # Need to run in parallel
    my $z = power($x,$y);
    my $t1 = Benchmark->new;
    my $t = timediff($t1,$t0)->real();

    $self->render(text => "n=$n, x=$x, y=$y, z=$z;<br>T=$t seconds");
};

app->start;

In other words, I'd like to reduce the time it takes to run down to 2 seconds (instead of 3) by running (add1 & mult2) in parallel.

This thread uses a Mojo::IOLoop->timer which doesn't seem relevant in my case? If so, I don't know how to use it. Thanks!

Answer 1

To avoid long waiting times, you can use the Mojolicious non-blocking operations. Instead of running a synchronous request to an external system, use non-blocking methods that instead run some callback upon completion. E.g. to avoid a sleep, we would use Mojo::IOLoop->timer(...).

Here is a variant of your code that uses non-blocking operations, and uses Promises to properly sequence the functions:

use Mojolicious::Lite;
use Benchmark qw(:hireswallclock);

# example using non-blocking APIs
sub add1 {
    my ($a) = @_;
    my $promise = Mojo::Promise->new;
    Mojo::IOLoop->timer(1 => sub { $promise->resolve($a + 1) });
    return $promise;
}

# example using blocking APIs in a subprocess
sub mult2 {
    my ($b) = @_;
    my $promise = Mojo::Promise->new;
    Mojo::IOLoop->subprocess(
        sub {  # first callback is executed in subprocess
            sleep 1;
            return $b * 2;
        },
        sub {  # second callback resolves promise with subprocess result
            my ($self, $err, @result) = @_;
            return $promise->reject($err) if $err;
            $promise->resolve(@result);
        },
    );
    return $promise;
}

sub power {
    my ($x, $y) = @_;
    my $result = Mojo::Promise->new;
    Mojo::IOLoop->timer(1 => sub { $result->resolve($x ** $y) });
    return $result;
}

any '/' => sub {
    my ( $self ) = @_;

    my $n = int(rand(5));

    my $t0 = Benchmark->new;
    my $x_promise = mult2($n);
    my $y_promise = add1($n);
    my $z_promise = Mojo::Promise->all($x_promise, $y_promise)
        ->then(sub {
            my ($x, $y) = map { $_->[0] } @_;
            return power($x, $y);
        });
    Mojo::Promise->all($x_promise, $y_promise, $z_promise)
        ->then(sub {
            my ($x, $y, $z) = map { $_->[0] } @_;
            my $t1 = Benchmark->new;
            my $t = timediff($t1, $t0)->real();

            $self->render(text => "n=$n, x=$x, y=$y, z=$z;\nT=$t seconds\n");
        })
        ->wait;
};

app->start;

This is a lot more complex than your code, but completes within two seconds instead of three seconds, and does not block other requests that happen at the same time. So you could request this route thirty times at once, and get thirty responses two seconds later!

Note that Promise->all returns a promise with the values of all awaited promises, but puts the values of each promise into an array ref so we need to unpack those to get the actual values.

If you cannot use non-blocking operations, you can run the blocking code in a subprocess, using Mojo::IOLoop->subprocess(...). You can still coordinate the data flow through promises. E.g. see the above mult2 function for an example.