Search code examples
perlsplitincrementlarge-filesfilesplitting

Perl - Changing file name in the middle of write


I am trying to take a very large txt file (over a million lines) that I created in Perl and run it through a different statement in Perl that will essentially look something like this (note the following is shell)

a=0
b=1
while read line;
do
    echo -n "" > "Write file"${b}
    a=($a + 1)
    while ( $a <= 5000)
    do
        echo $line >> "Write file"${b}
        a=($a + 1)
    done
    a=0
    b=($b + 1)
done < "read file"

Trying to size it down to 5k lines per file, and incrementing each time (filename1.txt, filename2.txt, filename3.txt, etc)
This doesn't seem to work in shell, possibly due to the size of the input file, and for the life of me I can't think of how to change what file I am writing to in the middle of the loop..


Solution

  • As an aside, this is your fixed script:

    #!/bin/sh
    a=0
    b=1
    while read line; do
        if [ $a -eq 0 ]; then
            echo -n '' > out-file-${b}
        fi
    
        echo $line >> out-file-${b}
    
        a=$(( $a + 1 ))
        if [ $a -eq 10 ]; then
            a=0
            b=$(( $b + 1 ))
        fi
    done < in-file
    

    Tested with bash and dash.