Search code examples
windowspostgresqlubuntuencodingiconv

iconv on windows ubuntu subsystem


I downloaded a huge csv file (7,98 Gio) in order to import it on a postgres database. The problem is that the file is encoded in ISO-8859 and if I want to import it on postgres it must be in UTF-8.

So i tried to convert it in utf-8 using iconv command on Ubuntu subsystem (integrated in Windows 10). The problem is that the output file is still empty according to Properties window of the output file. And the command won't terminate until Ctrl+C is pressed.

Here is my command :

iconv -t utf-8 < sirc-17804_9075_14209_201612_L_M_20170104_171522721.csv > xaus.csv

I've tried many syntaxes but none of theme are populating the output file...

P-S sorry for my english I'm french

edit : after a very long period the commands outputs :

iconv: unable to allocate buffer for input: Cannot allocate memory

Solution

  • iconv appears to want to load the entire file into memory, which may be problematic for large files. See iconv-chunks for a possible solution; from the iconv-chunks description:

    This script is just a wrapper that processes the input file in manageable chunks and writes it to standard output.