Search code examples
hadoopapache-pig

Analysing log files using Pig

I am new to Pig Latin and I need to analyze a log file to check for error logs and warning logs.

How to do that ? What is the script for finding errors and warnings ?

Solution

  • You would start off by LOAD-ing each line of a file as a string, then applying and extracting a regex pattern from each line, followed by a FILTER for errors and warnings.

    Elasticsearch is more commonly used for log analysis instead of Hadoop tools, though.