I'm attempting to upgrade Neo4j 3.5.5 to version 3.5.11 using the new APOC environment variable.
The upgrade works on my dev box running Ubuntu 18.04 via:
docker run -d -p 7474:7474 -p 7687:7687 -v /var/lib/neo4j/data:/data -v /var/lib/neo4j/plugins:/var/lib/neo4j/plugins -v /var/lib/neo4j/logs:/var/log/neo4j --ulimit=nofile=40000:40000 --env=NEO4J_AUTH=none -e NEO4J_dbms_allow__upgrade=true -e NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* --env 'NEO4JLABS_PLUGINS=["apoc", "graph-algorithms"]' neo4j:3.5.11
However it fails on the production platform, an EC2 instance running as a worker node in a Docker swarm, launched in a container built from node:10.16.0-alpine. Earlier versions up to and including 3.5.5 will install without error and have run successfully in this configuration for several years. The graph.db is identical on dev and production platforms. The compose file is as follows, where the only mods are the neo4j version number and the added NEO4JLABS_PLUGINS variable:
version: "3.2"
services:
neo4j:
image: neo4j:3.5.11
environment:
- HOME=/root
- NEO4J_AUTH=none
- NEO4J_dbms_allow__upgrade=true
- NEO4J_dbms_memory_pagecache_size=100M
- NEO4J_dbms_memory_heap_initial__size=2G
- NEO4J_dbms_memory_heap_max__size=2G
- NEO4J_dbms_security_procedures_unrestricted=apoc.\\\*
- NEO4JLABS_PLUGINS=["apoc", "graph-algorithms"]
ports:
- "7474:7474"
- "7687:7687"
volumes:
- gc01_data:/data
- gc01_neo4j_logs:/logs
networks:
- backend
ulimits:
nofile:
soft: 65535
hard: 65535
deploy:
replicas: 1
placement:
constraints: [engine.labels.node_task == neo4j]
restart_policy:
condition: on-failure
secrets:
gcrt_cert:
external: true
gcrt_key:
external: true
networks:
frontend:
backend:
volumes:
nmod_core:
gc01_neo4j_logs:
gc01_data:
external: true
The error log (from CloudWatch) looks like this:
17:33:12
Fetching versions.json for Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/raw/master/versions.json
Installing Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/3.5.0.5/apoc-3.5.0.5-all.jar to /var/lib/neo4j/plugins/apoc.jar
Fetching versions.json for Plugin 'graph-algorithms' from https://github.com/neo4j-contrib/neo4j-graph-algorithms/raw/master/versions.json
Installing Plugin 'graph-algorithms' from https://s3-eu-west-1.amazonaws.com/com.neo4j.graphalgorithms.dist/neo4j-graph-algorithms-3.5.11.0-standalone.jar to /var/lib/neo4j/plugins/graph-algorithms.jar
Active database: graph.db
Directories in use:
home: /var/lib/neo4j
config: /var/lib/neo4j/conf
logs: /logs
plugins: /var/lib/neo4j/plugins
import: /var/lib/neo4j/import
data: /var/lib/neo4j/data
certificates: /var/lib/neo4j/certificates
run: /var/lib/neo4j/run
Starting Neo4j.
2019-10-18 17:33:25.050+0000 INFO ======== Neo4j 3.5.11 ========
2019-10-18 17:33:25.109+0000 INFO Starting...
2019-10-18 17:33:25.576+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@192d43ce' was successfully initialized, but failed to start. Please see the attached cause exception "/logs/debug.log (Permission denied)". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@192d43ce' was successfully ini
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@192d43ce' was successfully initialized, but failed to start. Please see the attached cause exception "/logs/debug.log (Permission denied)".
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@192d43ce' was successfully initialized, but failed to start. Please see the attached cause exception "/logs/debug.log (Permission denied)".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
... 3 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /logs/debug.log (Permission denied)
at org.neo4j.graphdb.factory.module.PlatformModule.createLogService(PlatformModule.java:324)
at org.neo4j.graphdb.factory.module.PlatformModule.<init>(PlatformModule.java:181)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.createPlatform(GraphDatabaseFacadeFactory.java:263)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:180)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:148)
at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: java.io.FileNotFoundException: /logs/debug.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.openAsOutputStream(DefaultFileSystemAbstraction.java:72)
at org.neo4j.io.file.Files.createOrOpenAsOutputStream(Files.java:51)
at org.neo4j.logging.RotatingFileOutputStreamSupplier.openOutputFile(RotatingFileOutputStreamSupplier.java:338)
at org.neo4j.logging.RotatingFileOutputStreamSupplier.<init>(RotatingFileOutputStreamSupplier.java:137)
at org.neo4j.logging.RotatingFileOutputStreamSupplier.<init>(RotatingFileOutputStreamSupplier.java:121)
at org.neo4j.logging.internal.StoreLogService.<init>(StoreLogService.java:181)
at org.neo4j.logging.internal.StoreLogService.<init>(StoreLogService.java:45)
at org.neo4j.logging.internal.StoreLogService$Builder.build(StoreLogService.java:125)
at org.neo4j.graphdb.factory.module.PlatformModule.createLogService(PlatformModule.java:320)
... 12 more
2019-10-18 17:33:25.594+0000 INFO Neo4j Server shutdown initiated by request
Anything stand out? I'm seeing "Permission Denied" errors I don't understand. Thanks!
Yesterday I had the same problem, same error message, as the OP in 2019. Pretty weird to realize the OP was me in a past life.
Anyway, I realized the "home" directory for current versions of neo4j is at /var/lib/neo4j. This is the default, or may be set explicitly via env var NEO4J_HOME. It's displayed in the neo4j logs, as above where it says Directories in Use: home: /var/lib/neo4j.
Back then I was mounting the volume containing the neo4j data in a compose file as above - gc01_data:/data. Now I'm using a Docker CLI run option as -v /var/lib/neo4j/data:/data. In both cases the result is that data is mounted inside the container at /data, when neo4j is looking at /var/lib/neo4j/data. The error reads as "Permission Denied" but the real problem is "file not found." The mount target for the data volume has to match the neo4j home, as in: -v /var/lib/neo4j/data:/var/lib/neo4j/data