Search code examples
hadoopapache-pigbigdata

Invalid scalar projection in foreach

Hi I have a pig script like this. When doing foreach statement it throws invalid scalar projection error.Here is my code.

a = load 'file' using PigStorage(':');
b = group a by ($1, $7, $11);
c = foreach b generate flatten(group), COUNT(a) as (cnt: int);
d = filter c by cnt>1;
e = foreach d generate flatten(a) ;

The error is shown below

<line 6, column 31> Invalid scalar projection: a : A column needs to be projected from a relation for it to be used as a scalar

Any help will be appreciated.

Solution

  • The issue is because, 'a' doesn's exists in the 'd' relation schema.

    describe the 'd' schema, you get: d: {bytearray,bytearray,bytearray, cnt: int} in which 'a' doesn't exists.

    In the script, C relation is formed, by Projection of flattening the group field and Number of elements of a, a is not included in the relation C.