Search code examples
importhivesqoophcatalog

Permission issue encountered during sqoop import using Hcatalog

I am trying to use sqoop import with HCatalog integration to ingest data from Teradata to Hive. Below is my sqoop import command:

sqoop import -libjars /path/tdgssconfig.jar \
         -Dmapreduce.job.queuename=${queue} \
         -Dmapreduce.map.java.opts=-Xmx16g \
         -Dmapreduce.map.memory.mb=20480 \
         --driver com.teradata.jdbc.TeraDriver \
         --connect jdbc:teradata:<db-url>,charset=ASCII,LOGMECH=LDAP \
         --username ${srcDbUsr} \
         --password-file ${srcDbPassFile} \
         --verbose \
         --query "${query} AND \$CONDITIONS" \
         --split-by ${splitBy} \
         --fetch-size ${fetchSize} \
         --null-string '\\N' \
         --null-non-string '\\N' \
         --fields-terminated-by , \
         --hcatalog-database ${tgtDbName} \
         --hcatalog-table ${tgtTblName} \
         --hcatalog-partition-keys ${partitionKey} \
         --hcatalog-partition-values "${partitionValue}"

And I encountered below error - Error adding partition to metastore. Permission denied.:

18/07/03 12:14:02 INFO mapreduce.Job: Job job_1530241180113_6487 failed with state FAILED due to: Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : org.apache.hadoop.security.AccessControlException: Permission denied. user=<usr-name> is not the owner of inode=<partition-key=partition-value>
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:195)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:181)
    at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:178)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3560)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3543)
    at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3508)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6559)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1807)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1787)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:654)
    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setPermission(AuthorizationProviderProxyClientProtocol.java:174)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)

    at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:969)
    at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:249)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

How can I resolve this permission issue?

Solution

  • Figured out the issue. sqoop hcatalog cannot add files to Hive internal table, because it resides in Hive directories and the owner is hive, not a particular user. Resolution is to create an external table so that the underlying directories have the user (not hive) as the owner.