Search code examples
hiveapache-nifihortonworks-data-platformhdf

Nifi PutHiveStreaming writing data to HDFS but not able to see data in Hive table

  1. I'm using HDF Nifi 1.5 and Hdp Hive 3.1.

  2. Create a partioned and bucketing table in Hive:

CREATE TABLE default.client_tbl ( client_key int, clt_name varchar(45), clt_description varchar(200), version int ) partitioned by (clt_status varchar(8)) CLUSTERED BY(client_key) INTO 3 BUCKETS STORED AS ORC TBLPROPERTIES('transactional'='true');

  1. Used ExecuteSQL --> PutHiveStreaming

Nifi log shows the data written to HDFS path but I'm not able to view data in Hive table.

Please find the attached for nifi-app.log.

enter image description here

Solution

  • If you are using HDP Hive 3.1, then you will want PutHive3Streaming rather than PutHiveStreaming. The former is for Hive 3, the latter is for Hive 1.2.x.

    Having said that, the Hive 3 components are available as of NiFi 1.7.0+. Also if you are using an Apache NiFi distribution, it does not include the Hive 3 NAR by default (due to its size). You would have to build it yourself or use the one built for HDF (for use against HDP Hive 3), you can get the HDF 3.2 version (NiFi 1.7.0) version here.