Search code examples
kibana-4

Where is the kibana error log? Is there a kibana error log?

QUESTION: how do I debug kibana? Is there an error log?

  • PROBLEM 1: kibana 4 won't stay up
  • PROBLEM 2: I don't know where/if kibana 4 is logging errors

DETAILS: Here's me starting kibana, making a request to the port, getting nothing, and checking the service again. The service doesn't stay up, but I'm not sure why.

vagrant@default-ubuntu-1204:/opt/kibana/current/config$ sudo service kibana start
kibana start/running, process 11774

vagrant@default-ubuntu-1204:/opt/kibana/current/config$ curl -XGET 'http://localhost:5601'
curl: (7) couldn't connect to host

vagrant@default-ubuntu-1204:/opt/kibana/current/config$ sudo service kibana status
kibana stop/waiting

Here's the nginx log, reporting when I curl -XGET from port 80, which is forwarding to port 5601:

2015/06/15 17:32:17 [error] 9082#0: *11 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: kibana, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5601/", host: "localhost"

UPDATE: I may have overthought this a bit. I'm still interested in ways to view the kibana log, however! Any suggestions are appreciated!

I've noticed that when I run kibana from the command-line, I see errors that are more descriptive than a "Connection refused": 

vagrant@default-ubuntu-1204:/opt/kibana/current$ bin/kibana
{"@timestamp":"2015-06-15T22:04:43.344Z","level":"error","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.<anonymous> (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
{"@timestamp":"2015-06-15T22:04:43.346Z","level":"fatal","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.<anonymous> (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/usr/local/kibana-4.0.2/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
vagrant@default-ubuntu-1204:/opt/kibana/current$
Solution

  • Kibana 4 logs to stdout by default. Here is an excerpt of the config/kibana.yml defaults:

    # Enables you specify a file where Kibana stores log output.
# logging.dest: stdout

    So when invoking it with service, use the log capture method of that service. For example, on a Linux distribution using Systemd / systemctl (e.g. RHEL 7+):

    journalctl -u kibana.service

    One way may be to modify init scripts to use the --log-file option (if it still exists), but I think the proper solution is to properly configure your instance YAML file. For example, add this to your config/kibana.yml:

    logging.dest: /var/log/kibana.log

    Note that the Kibana process must be able to write to the file you specify, or the process will die without information (it can be quite confusing).

    As for the --log-file option, I think this is reserved for CLI operations, rather than automation.