When running a backup on Neo4j ent 3.0.5 it fails with the following:
2017-11-03 14:52:43.512+0000 INFO [o.n.c.s.StoreCopyClient] Done, copied 114
files
2017-11-03 14:52:43.923+0000 INFO [o.n.b.BackupService] Start recovering
store
command failed: Backup failed: Unexpected Exception
When I check /var/log/neo4j/debug.log it seems to think it's all OK.
2017-11-03 14:52:07.423+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Starting check pointing...
2017-11-03 14:52:07.423+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Starting store flush...
2017-11-03 14:52:08.260+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Store flush completed
2017-11-03 14:52:08.260+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Starting appending check point
entry into the tx log...
2017-11-03 14:52:08.303+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Appending check point entry
into the tx log completed
2017-11-03 14:52:08.303+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by full backup [2012424]: Check pointing completed
2017-11-03 14:52:11.582+0000 INFO [o.n.b.BackupImpl] BackupServer:6362-6:
Full backup finished.
I'm running the following commands:
sudo mkdir -m +x -p /backuptoazure
sudo mount -t cifs //xxxx.file.core.windows.net/neo4jbackup /backuptoazure -o vers=3.0,username=xxxx,password=xxxx,dir_mode=0777,file_mode=0777,sec=ntlmssp
sudo neo4j-admin backup --backup-dir=/backuptoazure --name="graph.db-backup.$(date +%Y-%m-%d)"
/backuptoazure is a mountpoint located on Azure files
Points to note;
-When I execute the backup command the files are written to Azure into a temp-copy file (i.e it has access) but the files are removed after the failure
-If I run the backup command to a local FS (/tmp) it completes without issue
-I can CP a backup to the mountpoint proving it has access
Any suggestions on where to go next would be appreciated. In particular what is Neo4j when it Start recovering store would be interesting to know
After some digging it seems that during the backup process neo4j starts an instance of the backed up database on the CIFS mount point and in doing so it is trying to use a 'byte range lock' which my assumption is that it is not supported on Azure Files.
Adding the option 'nobrl' to the mount options does the following:
Do not send byte range lock requests to the server. This is necessary for certain applications that break with cifs style mandatory byte range locks (and most cifs servers do not yet support requesting advisory byte range locks).
So the final mount command should be:
sudo mount -t cifs //xxxx.file.core.windows.net/neo4jbackup /backuptoazure -o vers=3.0,username=xxxx,password=xxxx,dir_mode=0777,file_mode=0777,sec=ntlmssp,nobrl
This should work. While not listed explicitly, for more information on the restrictions of Azure Files, please see: