Search code examples
batch-filebatch-processing

How to read and edit (>3GB) csv file using Windows Batch?

I am using the below FOR-IN-DO syntax to read and get the last row of (>3GB) csv files

For /F "UseBackQ Delims==" %%A In (test.csv) Do Set "lastline=%%A"

The code works for small files. However, the FOR-IN-DO cannot open the large csv files. Is there a workaround for this problem?

Solution

  • Depending upon the content of your source file, i.e. the CSV is not TAB delimited, or contain TAB characters you need to preserve, you may be able to do it like this:

    Set "SourceFile=test.csv"
For /F %%G In ('%SystemRoot%\System32\find.exe /C /V "" 0^<"%SourceFile%"') Do Set /A "TotalLines=%%G-1"
For /F Delims^=^ EOL^= %%G In ('%SystemRoot%\System32\more.com +%TotalLines% 0^<"%SourceFile%"') Do Set "LastLine=%%G"

    Based upon your comment, I'd offer the following modification of the above:

    Set "SourceFile=%UserProfile%\Desktop\test.csv"
Set "DestinationDir=%UserProfile%\Documents"
Set "LineMatch=Specific String"
For /F %%G In ('%SystemRoot%\System32\find.exe /C /V "" 0^<"%SourceFile%"') Do Set /A "TotalLines=%%G-1"
For /F Delims^=^ EOL^= %%G In ('%SystemRoot%\System32\more.com +%TotalLines% 0^<"%SourceFile%"') Do If /I "%%G" == "%LineMatch%" Move /Y "%SourceFile%" "%DestinationDir%"