Combining logs from a cluster architecture

Question

I've been looking for solutions to combine logs from a load balanced architecture so that I could effectively debug a server error or an access log. As of now I need to check each web node individually. Any out of the box solutions would be nice.

Answer 1

There are many OOTB tools -- commercial and FOSS -- that will collect your logs and give an interface to visualize and search them. For FOSS, have a look at logstash and fluentd, which you can self-host (if you're not ready to off-site your logging).

You've thus far probably been grepping a log file on each server. A simple improvement to this is a little script that does it in parallel across machines. I occasionally do this with a "telegrep" script that is called like:

% TG_REMOTES=app1:app2:app3
% telegrep somepattern /path/to/monster.log.gz

The script essentially passes the pattern and file to ssh to loop over all the TG_REMOTES, like:

for svr in TG_REMOTES; do ssh $svr "zgrep --some-options... $pattern $logfile"; done