Search code examples
arangodbwal

ArangoDB timeout due to too many write ahead logs


I am trying to restart my ArangoDB instance but I keep getting a timeout and I think it is because of the replaying of the WAL log files.

At first there were 2725 files, now there are 2701. I've let Arango replay all the files (as seen below) but I still get a timeout.

2018-11-09T10:30:11Z [2285] INFO replaying WAL logfile '/var/lib/arangodb3/journals/logfile-2668165691.db' (2700 of 2701)
2018-11-09T10:30:11Z [2285] INFO replaying WAL logfile '/var/lib/arangodb3/journals/logfile-2668552250.db' (2701 of 2701)
2018-11-09T10:30:11Z [2285] INFO WAL recovery finished successfully

When I restart, the service hangs here:

2018-11-09T10:41:34Z [2233] INFO using storage engine mmfiles
2018-11-09T10:41:34Z [2233] INFO {syscall} file-descriptors (nofiles) hard limit is 131072, soft limit is 131072
2018-11-09T10:41:34Z [2233] INFO Authentication is turned on (system only), authentication for unix sockets is turned on

Two questions:

  • What are my options here to recover my data and restart arango?
  • Why is the number of log files only lowered with 24 while I replayed all of them?

Solution

  • I moved the log files to another directory and then I could restart ArangoDB fine. Beware, I think this results in data loss.