Search code examples
postgresqlcopybulkinsertnpgsql

How to ignore duplicates records of a large table when using Copy Binary in postgres

I am trying to do bulk insert data in postgres using Binary copy, as below

await connection.BeginBinaryImportAsync($"COPY ph_numbers FROM STDIN (FORMAT BINARY));

This table already contains millions of records and I just want to ignore the records already present there.

I went through the documentation and could not find any efficient way to ignore duplicates.

Solution

  • There is no way to ignore duplicates.

    You can COPY the data to a temporary table and then use INSERT ... SELECT ... ON CONFLICT ... DO NOTHING to merge the data into your existing table.