Search code examples
apache-pig

Printed HashTable in Pig-Latin?


For example, I have this data:

x, 23
y, 492
v, 2034
x, 45
z, 25
v, 29

Which I want to transform into:

x, 23, 45
y, 492
v, 2034, 29
z, 25

It would be the equivalent of a printed hash table.

Here is my current script:

logs = LOAD 'tmp' using MyLoader (Parameters) as 
       (x:bytearray, y:bytearray, z, x1, y1:bytearray, z1:long, x2:bytearray,  
       z2:bytearray, z3:bytearray, z4:float, dataMap:map[], 
       recs:bag{(record:bytearray)}, key:bytearray, colo:bytearray);

filtered_logs = foreach logs { 
    info = FILTER records BY record MATCHES 'FIRST_REGEX';
    info_records = FOREACH info GENERATE GET_FIELDS($0) as 
                   rec:tuple(mClass:bytearray, rType:bytearray, 
                   rName:bytearray, rStatus:bytearray, rDuration:float, 
                   rData:bytearray, rDataMap:map[]);

    name = FOREACH info_records GENERATE rec.rName;

    matching_requests = FILTER records BY record MATCHES 'SECOND_REGEX'; 

    GENERATE FLATTEN(client_name) as client_name:chararray, 
    dataMap#'corr_id_', (SIZE(matching_requests) > 0 ? true : false) 
    as matched:boolean;
}

A = FILTER filtered_logs BY matched; 

key_corr_id = foreach A generate (chararray) $1 as key, (chararray) $2 as corr_id;

id_group = group key_corr_id by key; -- ERROR thrown when this line is included.

STORE id_group into '$output' using 
org.apache.pig.piggybank.storage.CSVExcelStorage(, 'YES_MULTILINE');

The error being thrown:

java.lang.ClassCastException: org.apache.pig.data.DataByteArrayString cannot be cast to java.lang.String

Solution

  • No need to create a new relation and join.Just group by the key and dump the relation.

    key_corr_id = foreach A generate (chararray) $1 as key:chararray, (chararray) $2 as corr_id:chararray;
    id_group = group key_corr_id by key;
    dump id_group;
    

    Now if you don't want the tuples say for key x , {(23),(45)} but want the items separated like x,23,45 then add another step to use BagToString on the corr_id in the grouping like this

    final = foreach id_group generate key,BagToString(A.$1, ',');
    dump final;