I am using Elastic stack 5.1.
I had elasticsearch and kibana installed with xpack installed on them, and they were working. I also installed logstash and beats, in the required order of installation.
After installing beats, and after some changes in my company VPN and proxy, my elasticsearch and kibana stopped working, the service does not even start.
I removed x-pack which got the ES and kibana to work, but again when I install x-pack, ES and kibana stop working. What should I do?
The log says the xpack plugin can't be loaded:
[2017-01-26T11:34:12,963][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: AccessDeniedException[/etc/elasticsearch/x-pack/system_key];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.2.jar:5.1.2]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_121]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_121]
at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_121]
at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]
... 6 more
So I was able to fix this..
ls -al /etc/elasticsearch/x-pack/system_key -rw------- 1 root elasticsearch 128 Nov 14 16:21 /etc/elasticsearch/x-pack/system_key
chown elasticsearch /etc/elasticsearch/x-pack/system_key
Looks like the elasticsearch-plugin command installed the x-pack plugin as root and elasticsearch itself didn't have access to read this file.