How do I monitor elasticsearch's logs using Graylog?

I need to monitor elasticsearch's logs by using graylog. I setted up graylog but I am confused as to use which input type to monitor from log files of elasticserach.

Solution

Take a look at these instructions: https://gist.github.com/joschi/e5d50048ddbcef038df9c4527b653ea9

Download and extract Elasticsearch

Download logstash-gelf and json-simple into the ./lib directory of Elasticsearch:

cd /path/to/elasticsearch/
pushd ./lib
wget http://central.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1.1/json-simple-1.1.1.jar \
  http://central.maven.org/maven2/biz/paluch/logging/logstash-gelf/1.10.0/logstash-gelf-1.10.0.jar
popd

Add logstash-gelf appender to config/logging.yml:

# you can override this using by setting a system property, for example -Des.logger.level=DEBUG
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file, gelf
logger:
  # [...]
appender:
  # [...]
  gelf:
    type: biz.paluch.logging.gelf.log4j.GelfLogAppender
    Host: "udp:127.0.0.1"
    Port: 12201
    Facility: elasticsearch
    ExtractStackTrace: true
    FilterStackTrace: true
    IncludeFullMdc: true

Start Elasticsearch

Remarks

If Graylog is not running or the configured GELF host is unreachable, you will see the following error messages at startup. They can be ignored and are specific to the GELF appender (others might throw other exceptions or none at all):

[2016-06-22 16:31:46,451][INFO ][node                     ] [Jonothon Starsmore] version[2.3.2], pid[30390], build[b9e4a6a/2016-04-21T16:03:47Z]
[2016-06-22 16:31:46,462][INFO ][node                     ] [Jonothon Starsmore] initializing ...
log4j:ERROR null
java.io.IOException: Cannot send data to /127.0.0.1:12201
    at biz.paluch.logging.gelf.intern.sender.GelfUDPSender.sendDatagrams(GelfUDPSender.java:59)
    at biz.paluch.logging.gelf.intern.sender.GelfUDPSender.sendMessage(GelfUDPSender.java:49)
    at biz.paluch.logging.gelf.log4j.GelfLogAppender.append(GelfLogAppender.java:95)
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
    at org.apache.log4j.Category.callAppenders(Category.java:206)
    at org.apache.log4j.Category.forcedLog(Category.java:391)
    at org.apache.log4j.Category.log(Category.java:856)
    at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalInfo(Log4jESLogger.java:120)
    at org.elasticsearch.common.logging.support.AbstractESLogger.info(AbstractESLogger.java:81)
    at org.elasticsearch.node.Node.<init>(Node.java:151)
    at org.elasticsearch.node.Node.<init>(Node.java:140)
    at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Caused by: java.net.PortUnreachableException
    at sun.nio.ch.DatagramDispatcher.write0(Native Method)
    at sun.nio.ch.DatagramDispatcher.write(DatagramDispatcher.java:51)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.DatagramChannelImpl.write(DatagramChannelImpl.java:605)
    at biz.paluch.logging.gelf.intern.sender.GelfUDPSender.sendDatagrams(GelfUDPSender.java:56)
    ... 15 more