Search code examples
hadoophivesqoop

How to use Sqoop import command with --map-column-hive?

I'm trying to Sqoop the data from Teradata to hive. I thought of following the below steps:

1) Create a Hive table with all the required fields in Hue.
2) By using Sqoop import command along with --map-column-hive attribute to load the data from Teradata to hive .

How to point to the already created Hive table from Sqoop import command, so that the Sqooped data should be placed in the corresponding Hive table?

Solution

  • You can generate map-column-hive attribute from existing table using shell and awk. it will be generated in the form COL1=TYPE,COL2=TYPE,...COLN=TYPE

    #!/bin/bash

#Set table name here
TABLE_NAME=your_schema.your_table

#generate map from existing table
MAP_COLUMN_HIVE=$(hive -S -e "set hive.cli.print.header=false; describe ${TABLE_NAME};" | awk -F " " 'f&&!NF{exit}{f=1}f{printf c toupper($1) "=" toupper($2)}{c=","}')

#call sqoop with --map-column-hive parameter
#add other sqoop params
sqoop import [your sqoop params here] --map-column-hive "$MAP_COLUMN_HIVE" [more sqoop params]