Running a Cypher batch statement (CALL apoc.periodic.iterate() in a @Query annotation of a Neo4jRepository), I encountered repeatedly the following error situation after a couple of hours of computation. Neo4j 3.4.9, Enterprise Edition.
2018-11-20 13:09:50.297+0000 ERROR [o.n.b.t.p.ProtocolHandshaker] Fatal error occurred during protocol handshaking: [id: 0x7035fd8f, L:/127.0.0.1:7687 - R:/127.0.0.1:56319] Connection reset by peer
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1108)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Connection to the database terminated. This can happen due to network instabilities, or due to restarts of the database
at org.neo4j.driver.internal.util.ErrorUtil.newConnectionTerminatedError(ErrorUtil.java:45) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.util.ErrorUtil.newConnectionTerminatedError(ErrorUtil.java:38) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.async.inbound.ChannelErrorHandler.channelInactive(ChannelErrorHandler.java:76) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1010) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1409) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:927) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[neo4j-java-driver-1.6.3.jar:1.6.3-b33d284ef6f350edbfe9e8f4539cdcdb0200bd0b]
at java.lang.Thread.run(Thread.java:844) [?:?]
The application server as well as the Neo4j server are running on the same machine, so the error message assumption regarding "network instabilities" I asses as misleading.
What is the reason for the sudden loss of database connection to Neo4j, after everything works fine for hours? How can I fix the uncontrolled and hard termination of the Neo4j server?
If you also encounter the error scenario "Fatal error occurred during protocol handshaking: Connection reset by peer" in Neo4j, the following steps may help you to locate the origin:
bin/neo4j-admin check-consistency --database=graph.db more infoAt the end of the day a missing transaction log was probably the reason for a corrupted database, which was triggered by a full hard disc. I had to delete and build up the database from the scratch. Since then everything is working fine again.