Search code examples
hadoophivehdfsapache-pigbigdata

Merging Part file in PIG


How can we merge these n part files(output file from pig operation). I have to merge these files into one and then use that as an input in hive. I tried using -cat and -getmerge but no luck. Any suggestion will be helpful.

Thanks


Solution

  • The Hadoop FileUtil class has a copyMerge method that handles very well the multiple part files problem.

    It will take all files in the folder you provide (although will not look recursively) and concatenate them all to produce a single file.