Does anyone has experienced on using Custom mapper in Sqoop Export from Hive Table to SQLDatabase before?
I had using the following sqoop command with 33 mappers to performed sqoop export:
sqoop export -Dmapred.job.queue.name=projectname -Dsqoop.export.records.per.statement=1000 --connect "jdbc:sqlserver://svrname;database=dbname" --username 'usrname' --password 'pwd' --hcatalog-database hive_schema_name --hcatalog-table hive_obj_name --table 'SQL_DB_OBJ_NAME' -- --schema SQL_DB_SCHEMA_NAME --fields-terminated-by $'\x01' -m 33 -batch
But as a result, I can only see 4 mappers was using from application master and required a long time to complete due to Huge Amount of Data. So I'm wonder does anyone can help to confirmed whether custom mapper is able to used in sqoop export
Sqoop export supports number of mappers argument, but it will be ignored in your command. You have to move -- --schema <schema-name> to the end of command since the Sqoop CLI has the following structure:
sqoop TOOL PROPERTY_ARGS SQOOP_ARGS [-- EXTRA_ARGS]