Search code examples
hadoopapache-pig

Apache Pig - ORDER BY error java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer

I am trying to do ORDER BY in Apache Pig. It fails with error java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer

Attached is the part of the code 

  movies = LOAD 'ml-1m/movies.dat' Using PigStorage('\t') as (movieblob:chararray);
A = FOREACH movies GENERATE STRSPLIT (movieblob,'::',3) as movie1:(id:int,title:chararray,genre:chararray);
B = FOREACH A GENERATE movie1.id, movie1.title, TOKENIZE (movie1.genre,'|');
C = FOREACH B GENERATE id, title, FLATTEN ($2) as genre;
C_FLAT = FOREACH C GENERATE FLATTEN ($0) as id:int, FLATTEN ($1) as
title:chararray, FLATTEN ($2) as genre:chararray;
\de C_FLAT 
C_FLAT: {id: int,title: chararray,genre: chararray}
C_FLAT_ORDER = ORDER C_FLAT BY id;
\d C_FLAT_ORDER
2017-01-20 16:17:53,727 [main] ERROR org.apache.pig.tools.grunt.Grunt -   
ERROR 1066: Unable to open iterator for alias C_FLAT_ORDER. Backend error : 
java.lang.ClassCastException: java.lang.String cannot be cast to  
java.lang.Integer

The workaround that I tried and which worked is

store C_FLAT INTO 'ml-1m/movies/output' Using PigStorage('|');
m1 = LOAD 'ml-1m/movies/output' Using PigStorage('|') as
(id:int,title:chararray,genre:chararray);
order_m1 = ORDER m1 BY id;
grunt> \de m1
m1: {id: int,title: chararray,genre: chararray}
grunt> \d order_m1

Few lines from output

(3945,Digimon: The Movie (2000),Children's)
(3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)
(3947,Get Carter (1971),Thriller)
(3948,Meet the Parents (2000),Comedy)
(3949,Requiem for a Dream (2000),Drama)

I have tried 

C1 = FOREACH C GENERATE (int)id, title,genre;

and did order by id and it failed too. Any help will be appreciated.

Some data from Input file used - 'ml-1m/movies.dat'

 3945::Digimon: The Movie (2000)::Adventure|Animation|Children's
 3946::Get Carter (2000)::Action|Drama|Thriller
 3947::Get Carter (1971)::Thriller
 3948::Meet the Parents (2000)::Comedy
 3949::Requiem for a Dream (2000)::Drama
 3950::Tigerland (2000)::Drama
 3951::Two Family House (2000)::Drama
 3952::Contender, The (2000)::Drama|Thriller
Solution

  • If you are just trying to order the data, this is all you need

    movies = LOAD '/tmp/ml-1m/movies.dat' Using PigStorage('\t') as (movieblob:chararray);
A = FOREACH movies GENERATE FLATTEN(STRSPLIT(movieblob, '::', 3)) AS (id:int,title:chararray,genres:chararray);
B = ORDER A BY id;

    I think your issue is related to trying to get this

    (3946,Get Carter (2000),Action|Drama|Thriller)

    Into this

    (3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)

    In that case, referencing 

    inpt = LOAD '/tmp/ml-1m/movies.dat' Using PigStorage('\t') as (line:chararray);
splt = FOREACH inpt GENERATE FLATTEN(STRSPLIT($0, '::', 3));
movies = FOREACH splt GENERATE (int) $0 as id, (chararray)$1 as title,FLATTEN(STRSPLITTOBAG($2, '\\|', -1)) AS genre;
ordered_movies = ORDER movies BY id;

    \d ordered_movies produces

    (3945,Digimon: The Movie (2000),Adventure)
(3945,Digimon: The Movie (2000),Animation)
(3945,Digimon: The Movie (2000),Children's)
(3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)
(3947,Get Carter (1971),Thriller)
(3948,Meet the Parents (2000),Comedy)
(3949,Requiem for a Dream (2000),Drama)
(3950,Tigerland (2000),Drama)
(3951,Two Family House (2000),Drama)
(3952,Contender, The (2000),Drama)
(3952,Contender, The (2000),Thriller)

    I am still under the impression that a Spark Dataframe would be more useful.