Search code examples
postgresqlcsvimportcopy

PostgreSQL and import of huge CSV Files


I tried to import several .csv Files (8GB, 56GB and 11GB) and all of them failed with unknown error.

CREATE TABLE name(
    COMMIT_ID int,
    PARENT_ID int
);

COPY  FROM 'C:/Users/Public/name.csv' DELIMITER ',' NULL AS '\N' ESCAPE AS '\' CSV;

I tried over pgAdmin4 and cmd.

I use Windows 10 and PostgreSQL12

All other files that are from the same database and they are all <1GB. This files I can import without any issue


Solution

  • This might not answer the question directly but what I did is to downgrade my postgresql.

    I use 9.6.18 now and just used the normal command

    COPY  FROM 'C:/Users/Public/name.csv' DELIMITER ',' NULL AS '\N' ESCAPE AS '\' CSV;
    

    I know this is not the perfect solution. It just seems like it is an issue from postgresql and not an user issue