I have some CSV files that I would like to parse with grep (or some other function from terminal) in order to extract some informations. They are in this form:
* Comment 1
* Comment line 2 explaining what the following numbers mean
1000000 ; 3208105 ; 0.18 ; 0.45 ; 0.00015 ; 0.1485 ; 0.03 ; 1 ; 1 ; 5 ; 477003 ;
* Comment 3
* Comment 4 explaining the meaning of the following lines
* Comment 5
0; 706520; p; 30.4983
1; 20859; p; 57.8
2; 192814; p; 111.842
3; 344542; p; 130.543
4; 54605; p; 131.598
5; 64746; d; 140.898
6; 442082; p; 214.11
7; 546701; p; 249.167
8; 298394; p; 305.034
9; 81188; p; 305.034
.......
In each file there could be at most one line in which the third field is equal to d instead of p. So either there is a line containing a d or there is none.
I have a lot of files like this and what I would like to do is extract from each file the line (if present) containing the letter d and append after this line the last parameter of the first not-comment line, that in this example would be 47703.
Up to now I managed to extract separately the lines I need.
With this I can extract every line containing a d from every file I have:
grep -h -E ' d;' *.csv > output.csv
And with this I can extract exactly the number 47703 from a file like the one in the example:
grep -v -e "^*" -e " p; " -e " d; " example_file.csv | cut -d \; -f 11
But I don't know how to put these two together.
The final output I would like to obtain from the example at the beginning is a single line like this:
5; 64746; d; 140.898; 47703
and I would like to have a line like this for every CSV file in the current directory.
Is there a way to do this?
I used for loop to loop all .csv files and assign returned values from your greps to variables, that are concatenated at end of each loop echoed:
for f in *.csv ; do value=`grep -v -e "^*" -e " p; " -e " d; " -e '^\s*$' "$f" | cut -d \; -f 11` ; line=`grep -h -E ' d;' "$f" ; echo "$line;$value" ; done
Edit: (I also added -e '^\s*$' to first grep, that gets line with values on first not commented line. Before, it matched empty lines)
this only echoes lines like 5; 64746; d; 140.898; 47703, that you wanted. If you want to redirect it to some file (all found lines will be in single output file), you can add it to last echo in that long command, like:
for f in *.csv ; do value=`grep -v -e "^*" -e " p; " -e " d; " -e '^\s*$' "$f" | cut -d \; -f 11` ; line=`grep -h -E ' d;' "$f" ; echo "$line;$value" > output.csv ; done
for readability, same code on multiple lines:
for f in *.csv
do
value=`grep -v -e "^*" -e " p; " -e " d; " -e '^\s*$' "$f" | cut -d \; -f 11`
line=`grep -h -E ' d;' "$f"
echo "$line;$value"
done