I'm trying to read data from the ArangoDB Write Ahead Log (WAL) so that the data can be replicated into Kafka.
ArangoDB provides an API to do so but says the following about it:
As a public API it is only supported to access these REST endpoints on a single-server instance. While these APIs are also available on DBServer instances, accessing them as a user is not supported. This API replaces some of the APIs in
/_api/replication/.
Source: WAL Access API
There's also the Replication Logger API but the crucial /_api/replication/logger-follow endpoint has been deprecated.
Is there a way to read the WAL for ArangoDB clusters instead of just single-server instances?
As you already found out, at the time being we can't officilly support the replication API as an end user facing API in ArangoDB Clusters.
As the current implementation state defines, of the DB-Servers has its own WAL to tail. Depending on replication factor and sharding strategy you may see the same wal entries from several servers you're tailing at once. The cluster itself makes intensive use of the replication for sharding and resillience functionality.
Peeking into this without complete knowledge whats going on in there may lead to random side effects in your setup.
Replicating complete clusters is however possible as part of our commercial offering: https://docs.arangodb.com/3.11/deploy/arangosync/
If you have other ideas about what to do with the replication API and are interested in discussing this in more detail, reach out for me as willi via my official address.