Search code examples
hadoopapache-pighadoop2bigdata

What is the meaning of "unable to open iterator for an alias" in pig?

I was trying to use the union operator like as show below

uni_b = UNION A, B, C, D, E, F, G, H;

here all the relations A,B,C...H are having same schema

when ever I am using the dump operator, till 85% it running fine.. after that it is showing the following error..

ERROR 1066: Unable to open iterator for alias uni_b

what is this? where is the problem? how should I debug?

this is my pig script...

ip = load '/jee/jee_data.txt' USING PigStorage(',') as (id:Biginteger, fname:chararray , lname:chararray , board:chararray , eid:chararray , gender:chararray , math:double , phy:double , chem:double , jeem:double , jeep:double , jeec:double ,cat:chararray , dob:chararray);

todate_ip = foreach ip generate id, fname , lname , board , eid , gender   , math , phy , chem , jeem , jeep , jeec , cat , ToDate(dob,'dd/MM/yyyy') as dob;

jnbresult1 = foreach todate_ip generate id, fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, ROUND_TO(AVG(TOBAG( math , phy , chem )),3) as bresult, ROUND_TO(SUM(TOBAG(jeem , jeep , jeec )),3) as jresult , cat , dob;

rankjnbres = rank jnbresult1 by jresult DESC , bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

rankjnbres1 = rank jnbresult1 by bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

allper = foreach rankjnbres generate id, rank_jnbresult1 , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult, jresult , cat , dob , ROUND_TO(((double)((10000-rank_jnbresult1)/100.000)),3) as aper;

allper1 = foreach rankjnbres1 generate id, rank_jnbresult1 , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult, jresult , cat , dob , ROUND_TO(((double)((10000-rank_jnbresult1)/100.000)),3) as a1per;

SPLIT allper into cbseB if board=='CBSE', anbB if board=='Andhra Pradesh', apB if board=='Arunachal Pradesh', bhB if board=='Bihar', gjB if board=='Gujarat' , jnkB if board=='Jammu and Kashmir', mpB if board=='Madhya Pradesh', mhB if board=='Maharashtra',  rjB if board=='Rajasthan' ,  ngB if board=='Nagaland' ,  tnB if board=='Tamil Nadu' , wbB if board=='West Bengal' ,  upB if board=='Uttar Pradesh';

rankcbseB = rank cbseB by jresult DESC , bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

grp = group rankcbseB all;

maxno = foreach grp generate MAX(rankcbseB.rank_cbseB) as max1;

cbseper = foreach rankcbseB generate id, rank_cbseB , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult, jresult , cat , dob , ROUND_TO(((double)((maxno.max1-rank_cbseB)*100.000/maxno.max1)),3) as per , aper;

rankBcbseB = rank cbseB by bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

grp = group rankBcbseB all;

maxno = foreach grp generate MAX(rankBcbseB.rank_cbseB) as max1;

Bcbseper = foreach rankBcbseB generate id, rank_cbseB , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult, jresult , cat , dob , ROUND_TO(((double)((maxno.max1-rank_cbseB)*100.000/maxno.max1)),3) as bper , aper;

rankanbB = rank anbB by jresult DESC , bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

grp = group rankanbB all;

maxno = foreach grp generate MAX(rankanbB.rank_anbB) as max1;

anbper = foreach rankanbB generate id, rank_anbB , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult,jresult , cat , dob , ROUND_TO(((double)((maxno.max1-rank_anbB)*100.000/maxno.max1)),3) as per , aper;

rankBanbB = rank anbB by bresult DESC , jeem DESC, math DESC, jeep DESC, phy DESC, jeec DESC, chem DESC, gender ASC, dob ASC, fname ASC, lname ASC DENSE;

grp = group rankBanbB all;

maxno = foreach grp generate MAX(rankBanbB.rank_anbB) as max1;

Banbper = foreach rankanbB generate id, rank_anbB , fname , lname , board , eid , gender , math , phy , chem , jeem , jeep , jeec, bresult, jresult , cat , dob , ROUND_TO(((double)((maxno.max1-rank_anbB)*100.000/maxno.max1)),3) as bper , aper;

joinall = join cbseper by (per) , Bcbseper by (bper) ;

joinall = foreach joinall generate Bcbseper::id as id,cbseper::jresult as b1;

A = cross Bcbseper , allper;

A1 = foreach A generate Bcbseper::id as id,Bcbseper::rank_cbseB as rank,Bcbseper::fname as fname,Bcbseper::lname as lname,Bcbseper::board as board,Bcbseper::eid as eid ,Bcbseper::gender as gender, Bcbseper::bresult as bresult,Bcbseper::jresult as jresult,Bcbseper::cat as cat,Bcbseper::dob as dob,Bcbseper::bper as bper,Bcbseper::aper as aper,allper::jresult as b2,allper::aper as a1per;

B = filter A1 by bper > a1per;

C = group B by id;

Dcbse = foreach C {
E = order B by a1per DESC;
F = limit E 1;
generate FLATTEN(F.id) , FLATTEN(F.b2);
};

joincbse = join joinall by id , Dcbse by id;

joincbse = foreach joincbse generate joinall::id as id , joinall::b1 as b1, Dcbse::null::b2 as b2;

joinall = join anbper by (per) , Banbper by (bper) ;

joinall = foreach joinall generate Banbper::id as id,anbper::jresult as b1;

A = cross Banbper , allper;

A1 = foreach A generate Banbper::id as id,Banbper::rank_anbB as rank,Banbper::fname as fname,Banbper::lname as lname,Banbper::board as board,Banbper::eid as eid ,Banbper::gender as gender, Banbper::bresult as bresult,Banbper::jresult as jresult,Banbper::cat as cat,Banbper::dob as dob,Banbper::bper as bper,Banbper::aper as aper,allper::jresult as b2,allper::aper as a1per;

B = filter A1 by bper > a1per;

C = group B by id;

Danb = foreach C {
E = order B by a1per DESC;
F = limit E 1;
generate FLATTEN(F.id) , FLATTEN(F.b2);
};

joinanb = join joinall by id , Danb by id;

joinanb = foreach joinanb generate joinall::id as id , joinall::b1 as b1, Danb::null::b2 as b2;

uni_b = UNION joincbse , joinanb ;
Solution

  • I got the solution to this. What is did was the following...

    First i stored all the relations A, B, C, .... using store operation as follows

    STORE A into into '/opA/' using PigStorage(',');

    Then, I Loaded the input for all the relation using load operation as follows

    ipA = load '/opA/part-r-00000' USING PigStorage (',') as (id:Biginteger, b1: double, b2: double);

    And at last I did the union using the union operation as follows

    uni_b = UNION ipA ,ipB ,ipC , ipD ,ipE ;

    I got the answer without any error.