Search code examples
aerospike

Aerospike - “n_bytes_memory went negative” with in memory-only namespace with ttl


We have a namespace configured to store data in memory only with couple of minutes default ttl. After starting putting some data into it, when expiration kicks in, we're getting these messages in the log (a lot, for ~30% of expired records): WARNING (namespace): (namespace.c::762) set_id 1 - n_bytes_memory went negative!

I have simple client app with server config that can reproduce this: https://github.com/akkomar/aerospike-test (it's based on docker and is very easy to start)

Any advice what might be the reason?

Edit:

I checked this on versions 3.6.4, 3.7.0.1 and 3.7.4

Configuration file used for testing (from https://github.com/akkomar/aerospike-test/blob/master/etc/aerospike.conf):

service {
    user root
    group root
    paxos-single-replica-limit 1
    pidfile /var/run/aerospike/asd.pid
    service-threads 4
    transaction-queues 4
    transaction-threads-per-queue 4
    proto-fd-max 1024
}

logging {
    file /var/log/aerospike/aerospike.log {
        context any info
    }
    console {
        context any info
        context namespace detail
    }
}

network {
    service {
        address any
        port 3000
    }
    heartbeat {
        mode mesh
        port 3002
        mesh-port 3002
        interval 150
        timeout 10
    }
    fabric {
        port 3001
    }

    info {
        port 3003
    }
}

namespace test_ns {
  replication-factor 2
  memory-size 1G
  default-ttl 10S

  storage-engine memory
}

Edit2:

It seems that it's happening only if I update records via UDF. The simplest one that reproduces this:

local VAL_KEY = "v"

function add_data(rec, val_to_add, ttl_to_set)
    if aerospike:exists(rec) then
        rec[VAL_KEY] = val_to_add

        aerospike:update(rec)
    else
        rec[VAL_KEY] = val_to_add

        aerospike:create(rec)
    end

end

When I execute the same operation via Java API - everything seems to work fine (example github repo mentioned earlier is updated with Java API example)


Solution

  • It turned out it was a bug in Aerospike. It's fixed in version 3.7.4.1 (detailed explanation in https://discuss.aerospike.com/t/problem-with-expiring-records-in-memory-only-namespace-n-bytes-memory-went-negative/2560/6)