Upload multiple csv files using single COPY command in MonetDB

Question

I have multiple csv files in a directory that i want to upload to a table using COPY command in MonetDB.

We are using spark to generate multiple files from different executors in the same shared volume. So inserting them in different COPY commands can be expensive. Instead if there is way to just pass the directory and it uploads all the files in that or comma separated file names. Is it possible?

Answer 1

This should work:

COPY INTO tab FROM '/file/1', '/file/2', ... USING DELIMITERS ...

or

COPY INTO tab FROM 'file1', 'file2', ... ON CLIENT USING DELIMITERS ...

USING DELIMITERS is optional, it's just showing where the optional bits go.

This works as long as you don't have a build with assertions enabled. There is an incorrect assert in the code that triggers, but normal release builds should work.

I noticed that the documentation (https://www.monetdb.org/documentation-Aug2024/user-guide/sql-manual/data-loading/copy-from/) doesn't mention the possibility to have multiple files, so we need to fix that.