Search code examples
aerospike

Error on Write operation (code 22) after calling Truncate. - C# client

When I try to use Aerospike client Write() I obtain this error:
22 AS_PROTO_RESULT_FAIL_FORBIDDEN

The error occurs only when the Write operation is called after a Truncate() and only on specific keys. I tried to:

  • change the key type (string, long, small numbers, big numbers)
  • change the Key type passed (Value, long, string)
  • change the retries number on WritePolicy
  • add a delay (200ms, 500ms) before every write
  • generate completely new keys (GUID.NewGuid().ToString())

None solved the case so I think the unique cause is the Truncate operation.

The error is systematic; for the same set of keys fail exactly on the same keys.

The error occurs also when after calling the Truncate I wait X seconds and checking the Console Management the Objects number on the Set is "0" .

I have to wait minutes (1 to 5) to be sure that running the process the problem is gone.

The cluster has 3 nodes with replica factor of 2. SSD persistence

I'm using the NuGet C# Aerospike.Client v 3.4.4

Running the process on a single local node (docker, in memory) does not give any error.

How can I know when the Truncate() process (the delete operation behind it) is completely terminated and I can safely use the Set ?

[Solution]
As suggested our devops checked the timespan synchronization. He found that the NTP was not enabled on the machine images (by mistake).
Enabled it. Tested again. No more errors.

Thanks,

Alex

Solution

  • Sounds like a potential issue with time synchronization across nodes, make sure you have ntp setup correctly... That would be my only guess at this point, especially as you are mentioning it does work on a single node. The truncate command will capture the current time (if you don't specify a time) and will use that to prevent records written 'prior' to that time from being written. Check under the (from top of my head, sorry if not exactly this) /opt/aerospike/smd/truncate.smd to see on each node the timestamp of the truncated command and check the time across the different nodes.

    [Thanks @kporter for the comment. So the time would be the same in all truncate.smd file, but a time discrepancy between machine would then still cause writes to fail against some of the nodes]