Search code examples
csvsplit

CSV File too big, need to split it into smaller ones

I´m having trouble working with a big CSV File (4.000.000 rows aprox) and i´m looking for a way to divided such file into smaller ones.

Thank you for any ideas you could provide.

Solution

  • You didn't mention which operating system you're using. In UNIX/Linux systems, there's the split command, you can use it as follows to split a file into chunks of two lines:

    Prompt> split -l 2 file.txt testfile
Prompt> ls -ltra
...
-rwxrwxrwx 1 user user   64 Dec 22 15:47 testfileaa
-rwxrwxrwx 1 user user   64 Dec 22 15:47 testfileab
-rwxrwxrwx 1 user user    1 Dec 22 15:47 testfileac
...

    In case you're working with Windows, you might install a WSL app (Windows Subsystem for Linux), which installs a Linux-like program on your computer, giving you access to all the wonders of Linux commandline :-)

    As correctly mentioned by Robert, the second file does not contain the headers, but a simple head -n 1 testfile can handle this.