I've been trying to install Sensu for a while, I've been following the installation guide, everything was going well while installing, but when I was adding the example checks from the docs, Sensu suddenly couldn't connect to RabbitMQ. Specifically, when I added the check_cron.json config. I tried restarting and removing the config (and restarting again after that), but Sensu is consistently timing out. It's worth noting that RabbitMQ is installed and running, it's even possible to navigate trough the management interface, however Sensu doesn't seem to recognize that.
Below is my server log:
{"timestamp":"2014-07-03T11:39:00.254061-0300","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/check_cron.json","changes":{"checks":{"cron_check":[null,{"handlers":["default"],"command":"/etc/sensu/plugins/check-proc.rb -p crond -C 1","interval":30,"subscribers":["webservers"]}]}}}
{"timestamp":"2014-07-03T11:39:00.254279-0300","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"sensu-test","address":"127.0.0.1","subscriptions":["test","webservers"]}]}}
{"timestamp":"2014-07-03T11:39:00.262568-0300","level":"info","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"}
{"timestamp":"2014-07-03T11:39:00.265246-0300","level":"info","message":"loaded extension","type":"handler","name":"debug","description":"outputs json event data"}
{"timestamp":"2014-07-03T11:39:00.313486-0300","level":"info","message":"i am the master"}
{"timestamp":"2014-07-03T11:39:03.326913-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:39:03.327281-0300","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"2014-07-03T11:39:03.327463-0300","level":"warn","message":"resigning as master"}
{"timestamp":"2014-07-03T11:39:03.328908-0300","level":"info","message":"removed master lock"}
{"timestamp":"2014-07-03T11:39:11.343859-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:39:19.366786-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:39:20.294332-0300","level":"fatal","message":"rabbitmq connection error","error":"timed out while attempting to connect"}
{"timestamp":"2014-07-03T11:39:20.294816-0300","level":"warn","message":"stopping"}
{"timestamp":"2014-07-03T11:39:20.294959-0300","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"2014-07-03T11:39:20.295122-0300","level":"info","message":"completing handlers in progress","handlers_in_progress_count":0}
{"timestamp":"2014-07-03T11:39:20.796222-0300","level":"warn","message":"stopping reactor"}
{"timestamp":"2014-07-03T11:50:59.786233-0300","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"sensu-test","address":"127.0.0.1","subscriptions":["test","webservers"]}]}}
{"timestamp":"2014-07-03T11:50:59.808513-0300","level":"info","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"}
{"timestamp":"2014-07-03T11:50:59.811049-0300","level":"info","message":"loaded extension","type":"handler","name":"debug","description":"outputs json event data"}
{"timestamp":"2014-07-03T11:50:59.858109-0300","level":"info","message":"i am the master"}
{"timestamp":"2014-07-03T11:51:02.867913-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:51:02.868369-0300","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"2014-07-03T11:51:02.868466-0300","level":"warn","message":"resigning as master"}
{"timestamp":"2014-07-03T11:51:02.869157-0300","level":"info","message":"removed master lock"}
{"timestamp":"2014-07-03T11:51:10.889055-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:51:18.910561-0300","level":"warn","message":"reconnecting to rabbitmq"}
{"timestamp":"2014-07-03T11:51:19.842039-0300","level":"fatal","message":"rabbitmq connection error","error":"timed out while attempting to connect"}
{"timestamp":"2014-07-03T11:51:19.842408-0300","level":"warn","message":"stopping"}
{"timestamp":"2014-07-03T11:51:19.842551-0300","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"2014-07-03T11:51:19.842714-0300","level":"info","message":"completing handlers in progress","handlers_in_progress_count":0}
{"timestamp":"2014-07-03T11:51:20.343916-0300","level":"warn","message":"stopping reactor"}
If you are using ssl certificates on your rabbitmq server then you probably have a problem with the certificates. There is a script on github clone it and run the following commands:
git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git
cd joemiller.me-intro-to-sensu/
./ssl_certs.sh clean
./ssl_certs.sh generate
And then configure the Rabbitmq to use these certificates:
mkdir /etc/rabbitmq/ssl
cp server_key.pem /etc/rabbitmq/ssl/
cp server_cert.pem /etc/rabbitmq/ssl/
cp testca/cacert.pem /etc/rabbitmq/ssl/
And add this configuration to /etc/rabbitmq/rabbitmq.config:
[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
{keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].
here you tell rabbitmq which certificates it will be using, make sure the paths of the certificates are the same as the ones you copied the certificates to.
update-rc.d sensu-server defaults
update-rc.d sensu-api defaults
update-rc.d sensu-client defaults
update-rc.d sensu-dashboard defaults
And then copy the SSL client_key and certificate as you have done earlier using:
cp client_key.pem client_cert.pem /etc/sensu/ssl/
And then add the different configurations to /etc/sensu/config.json.