Search code examples
apache-pig

output pig fs command to a file or a variable

I am trying to monitor the files in my HDFS system. Currently, the way I am doing it is very painful: fs -ls -R /river/entity/files;.

This command output the result in the console log file. And then I need to copy/paste the result into a file to use it... which is not efficient at all. Using PIG, is there a simple way to output the result of the command directly into a file?

EDIT: Thx for the answers but I haven't been clear enough. Sorry ! I cannot use terminal. I execute only a pig script and I want my result in HDFS. Is it possible to do that only on hadoop side?

Solution

  • Yes you can ! In fact you can execute any shell command in your Pig script like this :

    %declare dummy `hdfs dfs -ls -R /river/entity/files | hdfs dfs -put - hdfs://nn.example.com/hadoop/myfile.log`

    But you should consider that Pig statements does not executed in the order that appear in your script !