Parse and plot data received from log-file on-the-fly

Question

I want to create a Perl (or Bash) script to create and plot data on-the-fly. That means I want to extract data from a log-file file.log, ideally don't write a tmp-file (if it's only possible with a tmp-file, this would be fine too) and plot with Gnuplot. As the logfile is growing, I want to see the additional information in the plot.

See this question for similar topic.

For Perl, what I have so far is this script:

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $switch = "off";

open(INFILE,"< $path") or die $! \n";
while (my $line = <INFILE>) {

       if ($line =~ m{^Time = (\d+)}){
               push(@grepped,"$1\t");
       };

       if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
               push(@grepped,"$1\t");
               push(@grepped,"$2\n");
               $switch = "refresh";
       };


if ($switch eq "refresh"){

open(GP, "| gnuplot -persist") or die: $! \n";
print GP << "GNU_EOF";

plot "@grepped" u 2:1
pause 1; refresh; reread;

GNU_EOF

close(GP);
}


}
close(INFILE)};

My first problem is that the on-the-fly functionality for Gnuplot isn't working. When the file.log gets changed, the refreshing of the Gnuplot-graph isn't working. I don't know if the -persist is correct here or if I have to use replot option instead of refresh. I tried it but it's not working for me.

Second problem is to read the data into array @grepped and use it in Gnuplot in one script for a changing input file. One additional restriction is that (as you can see from the code) gnuplot only should refresh the plot if a complete new line in @grepped is written. Otherwise errors would occur for sure because of the wrong assignment of data.

When I try a simple script like:

#!/usr/bin/gnuplot -persist
plot "data.dat" u 2:1
pause 1; replot; reread;

or

#!/usr/bin/gnuplot
plot "data.dat" u 2:1
pause 1; refresh; reread;

the on-the-fly part works if I change data.dat by hand and save it.

Answer 1

Here are two ways for plotting data on-the-fly.

Looping with gnuplot

You must call plot over and over again, the data is preprocessed by an external script. The minimal gnuplot script filter.gp is:

while (1) {
    plot '< ./myscript.pl' using 2:1
    pause 1
}

To stop this, hit Ctrl+C.

The Perl script for the preprocessing may look like the following myscript.pl:

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $t = 0;
open(INFILE,"< $path") or die "$! \n";

while (my $line = <INFILE>) {
    if ($line =~ m{^Time = (\d+)}){
        $t = $1;
    };

    if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
        print "$t\t$1\t$2\n";
    };
};
close(INFILE);

Just run it with gnuplot filter.gp.

To make it more configurable, one can change the script to use a variable which is passed to gnuplot via the command line:

while (1) {
    plot '< ./myscript.pl -f '.path using 2:1
    pause 1
}

or use reread for this:

plot '< ./myscript.pl -f '.path using 2:1
pause 1
reread

Call this script with gnuplot -e "path='file.log';" filtermod.gp.

That works, but would filters the complete file over and over again.

Piping from Perl to gnuplot

Here is a Perl script, which basically works for me, but its my first real Perl script, so there may be some nonideal parts. Feel free to comment on that.

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $switch = "off";

open(my $gp, "| gnuplot -persist") or die "$! \n";
$gp->autoflush(0);

use File::Tail;
my $file = File::Tail->new(name=>$path, maxinterval=>1, tail=>-1);

while (defined(my $line= $file->read)) {
    if ($line =~ m{^Time = (\d+)}){
        push(@grepped,"$1\t");
    };

    if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
        push(@grepped,"$1\t");
        push(@grepped,"$2\n");
        $switch = "refresh";
    };

    if ($switch eq "refresh") {
        print $gp <<"GNU_EOF";
plot '-' using 2:1
@grepped
e
GNU_EOF
        $gp->flush;
        $switch = "off"; 
    };
    Time::HiRes::sleep(0.1);
};

What I found to be important here is

1. Looping over a changing file.
2. The arrangement of the autoflush and flush.
3. The sleep to allow gnuplot to process the data properly.

That worked for a very small test data file. Don't know if it will for a larger one as well, but should help you a bit further.