Centos VM guest on Win10 host. Attempting to follow installation guide for x-pack (and step by step from the training course). ES and K running fine, but once x-pack plugin installed, kibana will start (and complains ES is missing - that's fine), then elastic search starts and immediately fails. It looks like the same error as 3, but in a local VM, not a production setting. Here's the tail of the log:
[2017-01-31T16:52:03,191][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.1.1.jar:5.1.1]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
... 14 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
... 14 more
[2017-01-31T16:52:03,215][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elkmaster1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: AccessDeniedException[/etc/elasticsearch/x-pack/system_key];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:384) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/x-pack/system_key
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_111]
at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:160) ~[?:?]
at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
at org.elasticsearch.xpack.security.Security.<init>(Security.java:191) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:193) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:373) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:132) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:294) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.node.Node.<init>(Node.java:229) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:214) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
... 6 more
Well since the log contains
Caused by: java.nio.file.AccessDeniedException:
The problem is possibly the user permissions. Who is the owner of this directory?
/etc/elasticsearch/x-pack
What does ls -l /etc/elasticsearch/x-pack give you for an output?
If you have sudo permissions or have root access, you might just change the ownership of this directory.