Search code examples
hadoopapache-pig

How to load a file in Pig that has a variable number of fields

I have an input file which contains records as follows:

Movie1     Actor 1, Actor 2, Actor 3, ......, Actor n
Movie2     Actor 1, Actor 2,.......Actor n

I want to load this data into a bag in Pig

 movies = LOAD 'movies.imdb' AS (......);

I am not sure how to fill up my "AS" field because the records in my input file may have variable number of fields.

Solution

  • You could just do this:

    movies = LOAD 'movies.imdb' USING PigStorage(',');

    If the records don't have fixed fields, using AS won't be possible