Search code examples
hadoopsqoopavro

How to pass column names having spaces to sqoop --map-column-java


I have to import data using sqoop, my source column names are having spaces in between them, so while I am adding it in --map-column-java parameter getting the error.

Sample Sqoop import:

sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java data col1=String, data col2=String, data col3=String --as-avrodatafile

Column names:

data col1,
data col2,
data col3

Error:

19/03/07 07:31:55 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1355) at org.apache.sqoop.SqoopOptions.setMapColumnJava(SqoopOptions.java:1375) at org.apache.sqoop.tool.BaseSqoopTool.applyCodeGenOptions(BaseSqoopTool.java:1363) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]*


Solution

  • Able to resolve this issue:

    1. Spaces issue: sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

    2. ERROR tool.ImportTool: Import failed: Cannot convert SQL type 2005: 3 columns in source are having 2005 and nvarchar added them in --map-column-java resolved this issue

    3. org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]: 1****

    This is causing due to using * in select query, so modified sqoop query as:

    sqoop import --connect jdbc-con --username "user1" --query "select [col1,data col2,data col3] from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile