Search code examples
gnuplot

Plotting data from two files after performing mathematical operation between them, with gnuplot for windows

something didn't go well... maybe it is the datetime format... the first file is like
01/11/2021 00:15:00 15.0 70.0 0.10 1010.0 0.8 228 1.4 0.0
01/11/2021 00:30:00 14.8 71.0 0.20 1010.0 1.0 200 1.9 0.0
01/11/2021 00:45:00 14.6 73.0 0.30 1010.0 0.8 142 1.4 0.0
01/11/2021 01:00:00 14.6 74.0 0.20 1010.0 1.2 147 2.0 0.0
and the second, like
01/11/2021 00:30 15,610 0,0 -1,257 18,15 60,54 0,153 69,74 32,81
01/11/2021 01:00 15,380 0,0 -1,008 18,03 62,31 0,400 120,60 25,78
(note that in this latter case, time does not have seconds); date and time format, I've written just before plotting the $Data, that is: set xdata time, set timefmt "%d/%m/%Y %H:%M" and set xrange ["01/11/2021 00:00":"01/11/2021 03:00"] ). Let's say I want to subtract column 5 (not counting the datetime columns) of text2, from column 2 of text 1. In this case, file1 has 8 colums (not counting datetime columns) and file2 too, but I might have more colums in file2 in the future. So it's all about subtracting data from two files only from the same timestamp, but one file has more timestamp data compared to the other one (though fortunately, x-entries of the 2nd file are cointained among the x-entries of the 1st file). Thank you!

Solution

  • Let me summarize what I understood from your modified question:

    • you have two files which have two different decimal separators: "." and ","

    • the two files have different timestamp formats: "%d/%m/%Y %H:%M" and "%d/%m/%Y %H:%M:%S"

    • the two files have different number of timestamps and different time intervals, but some timestamps are identical.

    • Now, you want to do a mathematical operation on two columuns from the two files, but only for those having identical timestamps.

    Data: (some more points added for illustration)

    'SO70191250_data1.dat'

    01/11/2021 00:15:00 15.0 70.0 0.10 1010.0 0.8 228 1.4 0.0
01/11/2021 00:30:00 14.8 71.0 0.20 1010.0 1.0 200 1.9 0.0
01/11/2021 00:45:00 14.6 73.0 0.30 1010.0 0.8 142 1.4 0.0
01/11/2021 01:00:00 14.6 74.0 0.20 1010.0 1.2 147 2.0 0.0
01/11/2021 04:00:00 11.1 74.0 0.20 1010.0 1.2 147 2.0 0.0
01/11/2021 04:44:44 11.1 74.0 0.20 1010.0 1.2 147 2.0 0.0
01/11/2021 06:00:00 22.2 74.0 0.20 1010.0 1.2 147 2.0 0.0

    'SO70191250_data2.dat'

    01/11/2021 00:30 15,610 0,0 -1,257 18,15 60,54 0,153 69,74 32,81
01/11/2021 01:00 15,380 0,0 -1,008 18,03 62,31 0,400 120,60 25,78
01/11/2021 02:00 15,380 0,0 -1,008 18,03 55,55 0,400 120,60 25,78
01/11/2021 04:00 15,380 0,0 -1,008 18,03 55,55 0,400 120,60 25,78
01/11/2021 04:22 15,380 0,0 -1,008 18,03 66,66 0,400 120,60 25,78
01/11/2021 06:00 15,380 0,0 -1,008 18,03 66,66 0,400 120,60 25,78

    Solution 1: use whatever programming language to prepare the data (i.e. modify decimal separator, find identical timestamps, do the mathematical operation, etc.) such that gnuplot can easily plot the final result.

    Solution 2: gnuplot-only. This will get a bit lengthy.

    First of all, the different decimal separators: You can set the following: (check help decimalsign)

    set decimalsign locale "French"   # set input/output decimalsign to comma ("German" and others should also work)
set decimalsign "."               # explicitely set output decimalsign to point (only with gprintf() command)

    With this you can change your "decimal comma" input file into a "decimal point" datablock.

    In the earlier version of the question you wrote that the time interval of the first file is half the interval of the second file, i.e. every second timestamp for the first file would correspond to a timestamp in the second file. For the following solution, this condition is not necessary it will do the desired mathematical operation only for those timestamps which are identical. They even don't have to be ordered. Only requirement is that there should not be two identical timestamps in the same file and the timestamps should not contain fractions of seconds.

    Overview about the general procedure:

    1. write your files with the columns of interest into datablocks $Temp1 and $Temp2 where the decimalsign of the second file will be changed from "," to ".".

    2. at the same time, change the different timestamps to the same format (here: seconds since the Unix epoch (1970-01-01 00:00) and additionally add a litte offset of 0.1 and 0.2 indicating the origin from the first and second file, respectively.

    3. merge the two datablocks $Temp1 and $Temp2 into the datablock $Temp3

    4. the gnuplot option smooth can be used to sort these timestamps into another datablock $Temp4. Hence, you will get the identical timestamps (with offset) and corresponding data in successive lines.

    Example of $Temp4:

    1635725700.1  15  i
1635726600.1  14.8  i
1635726600.2  60.54  i
1635727500.1  14.6  i
1635728400.1  14.6  i
1635728400.2  62.31  i
1635732000.2  55.55  i
1635739200.1  11.1  i
1635739200.2  55.55  i
1635740520.2  66.66  i
1635741884.1  11.1  i
1635746400.1  22.2  i
1635746400.2  66.66  i
    1. Create yet another datablock $Temp5 according to the following: if two successive lines have a timestamp difference of 0.1 (to be sure, check less than 1: t2-t1<1) then subtract the y values of these lines, otherwise assign NaN.

    Example of $Temp5:

    1635725700.0  NaN  u
1635726600.0  NaN  u
1635726600.0  45.74  i
1635727500.0  NaN  u
1635728400.0  NaN  u
1635728400.0  47.71  i
1635732000.0  NaN  u
1635739200.0  NaN  u
1635739200.0  44.45  i
1635740520.0  NaN  u
1635741884.0  NaN  u
1635746400.0  NaN  u
1635746400.0  44.46  i

    Code:

    ### column operation on two files with different timestamps and decimalsigns
reset session

FILE1 = 'SO70191250_data1.dat'
FILE2 = 'SO70191250_data2.dat'

# get datafile into datablock with unix time and offset
set table $Temp1
    plot FILE1 u (sprintf("%.1f",timecolumn(1,"%d/%m/%Y %H:%M:%S")+0.1)):3 w table
unset table

# modify decimalsign from comma to point with unix time and offset
set decimalsign locale "French"   # set input decimalsign to comma ("German" and other should also work)
set decimalsign "."               # set output decimalsign to point (works only with gprintf() command)
set table $Temp2
    plot FILE2 u (gprintf("%.1f",timecolumn(1,"%d/%m/%Y %H:%M")+0.2)):(gprintf("%g",$7)) w table
unset table
set decimalsign locale "English"

# append two datablocks
set print $Temp3
    do for [i=1:|$Temp1|] { print $Temp1[i] }
    do for [i=1:|$Temp2|] { print $Temp2[i] }
set print

# sort datablock by timestamp with offset
set table $Temp4
    set format x "%.1f"             # user 1 decimal format output
    plot $Temp3 u 1:2 smooth freq
unset table
print $Temp4

# make difference of y values only if difference in timestamp is <1
set table $Temp5
    plot t2=y2=NaN $Temp4 u (t1=t2,t2=$1,int($1)):(y1=y2,y2=$2, t2-t1<1 ? y2-y1 : NaN)
unset table
print $Temp5

set key top center
set format x "%d/%m\n%H:%M" time
set xtics 3600
set grid x,y
set datafile missing NaN

plot $Temp1 u (int($1)):2 w lp pt 7 lc "red" ti "File1", \
     $Temp2 u (int($1)):2 w lp pt 7 lc "web-green" ti "File2", \
     $Temp5 u 1:2 w lp pt 7 lc "blue" ti "Difference
### end of code

    Result: (difference will only be calculated for identical timestamps in the two files)

    enter image description here