Search code examples
bashterminalfile-manipulation

Appending file_name part to each line of the file, recursively to multiple files


So, I have multiple files of the form

filename_date.csv

in a folder, for example something like

filename_2010_01_01.csv

that files contains, say... 1000 rows with a format like this

37845287|2364532467|237849853
04568750|89345735385|94573857

But I have 365 files like this for the whole year, one for each day. What I'd like to do is to make some kind of bash? script maybe that takes the date from the filename and appends it to the end of each of the x lines in each file so the result would be something like

filename_2010_01_01.csv

37845287|2364532467|237849853|2010_01_01
04568750|89345735385|94573857|2010_01_01

filename_2010_01_02.csv

438574|563546567|2342523525|2010_01_02
326345452|456454763|7856763|2010_01_02

So far what I've tried:

I tried with this

sed 's/$/|2010-01-01/' filename_2010-01-01.csv > filename_2010-01-01.csv

that works for a single file.

Now I'm trying the following loop

FILES=~/Desktop/2010/byday/*
for f in $FILES
do
  filename=$f | cut -d'/' -f7-
done

Once I have the name of the file there, I can just add the sed command within the loop but I'm having trouble on making the filename var to take the value of the command

If I just echo the command within the loop it does get me the filename, that command is only to strip the path from the filename.

UPDATE:

Inside the loop

  filename=`basename $f`

that shorter, cleaner and the variable actually gets the value

UPDATE 2:

This loop

FILES=~/Desktop/2010/byday/*
for f in $FILES
do
  fname=`basename $f`
  fname=${fname%.csv}
  fname=${fname##filename_}
  echo $fname
  sed 's/$/|$fname/' filename_$fname.csv > filename__$fname.csv
done

Actually outputs correctly the date BUT there's something in the sed command that deletes the content of the file

UPDATE 3:

With this Im almost there

FILES=~/Desktop/2010/byday/*
for f in $FILES
do
  fname=`basename $f`
  fname=${fname%.csv}
  fname=${fname##partitioned.csvmerged_}
  sed 's/$/|$fname/' partitioned.csvmerged_${fname}.csv > $fname.csv
done

It writes a new file with just the date as filename (which is good) but Im not sure how should I interpolate the var there since with this the output inside the file is

438574|563546567|2342523525|$fname
326345452|456454763|7856763|$fname

Solution

  • You are overwriting the same file that you are reading. As soon as the shell parses the command, it starts truncating the file and then, with the output file prepared, it starts the sed command, that finds an empty file. Your problem will be solved by avoiding the redirection and using the -i parameter in sed:

    FILES=~/Desktop/2010/byday/*
    for f in $FILES
    do
        fname=$(basename "$f" .csv)
        fname=${fname##filename_}
        echo "$fname"
        sed -i "s/\$/|$fname/" "filename_$fname.csv"
    done
    

    I also added a few minor changes, like using the already present basename to get rid of the extension, quoting variables and solving your next problem: not having $fname expanded due to the use of single quotes.