I don't want to use pg_dump
to export data into sql script, since feeding it to the greenplum cluster is too slow when I have a large amount of data to import. So it seems using greenplum's gpfdist
is prefered. Is there any way I can do this?
Or as an alternative, can I export a particular Postgres table's data into a CSV format file containing the large orbjects of that table?
pg_dump will create a file that will use "COPY" to load the data back into a database. When loading into Greenplum, it will load through the Master server and for very large loads, it will become a bottleneck. Yes, the preferred method is to use gpfdist but you can most certainly use COPY to load data into Greenplum. It won't load in the 10+ TB per hour rate that gpfdist can achieve but it still can achieve 1 to 2 TB per hour.
Another alternative is to use gpfdist to execute a program to get data. It would execute the SELECT statement against PostgreSQL to make that available to an External Table in Greenplum. I created a wrapper for this process called, "gplink". You can check it out here: http://www.pivotalguru.com/?page_id=982