Search code examples
hiveapache-pighcatalog

Pig BigDecimal to Hive decimal

I am trying to convert Pig BigDecimal to Hive Decimal type but values are gettined null. Here is the sample code:

Pig Scripts:

    a = LOAD 'test.txt' using TextLoader() as (col1:chararray,col2:int,col3:chararray,col4:int);
    b = foreach a generate *,1 as rec_cnt;
    c = group b by col1,col3;
    d = foreach c generate flatten(group),(bigdecimal) SUM(rec_cnt) as grp_code;
STORE d into 'user/test' Using PigStorage(',');
STORE d into 'default.test' using org.apache.hive.hcatalog.pig.HCatStorer();

In the above code, the sum value of record counts is coming properly in the HDFS file stored as 'user/test'. But with HcatStorer, the same field is populated with NULL for all the records. The test table is created with this column definition of DECIMAL(16,0). I am using Hive 1.1.0. Please suggest how to fix this issue.

Solution

  • I have finally figured it out the reason for null. Hcatalog does a range check when converting bigdecimal in pig to Decimal in hive. As there is no scale in the Hive definition (i.e., DECIMAL (16,0)), it is getting defaulted to null during range check while storing. When I changed the Hive definition to DECIMAL(16,2) it is getting stored properly. So this requires a change in layout for sure to update the scale.