Can't remove record from aerospike

Question

I decided to try aerospike but I have some problems. I'm using aerospike in a docker:

companies-data:
    image: 'aerospike/aerospike-server:3.10.0-1'
    ports:
        - '5310:3000'
        - '5311:3001'
        - '5312:3002'
        - '5313:3003'
    volumes:
        - './companies-data/data:/opt/aerospike/data'
        - './companies-data/config:/opt/aerospike/etc'
    command: '/usr/bin/asd --foreground --config-file /opt/aerospike/etc/aerospike.conf'

When I create a record and then restart the docker container the data is still there, so volumes are set up correctly. However when I remove a record and restart the docker container then the record is still there, it's not removed. Before a restart it works fine: the record is removed but after docker-container restart it is there again.

I'm using nodejs aerospike client.

let key = new Key(this.ns, this.set, id);
client.remove(key, function (err, key) {
    if (err) {
        return reject(err);
    }
    resolve(key);
});

Here is my conf:

service {
    user root
    group root
    paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
    pidfile /var/run/aerospike/asd.pid
    service-threads 4
    transaction-queues 4
    transaction-threads-per-queue 4
    proto-fd-max 15000
}

logging {

    # Log file must be an absolute path.
    file /var/log/aerospike/aerospike.log {
        context any info
    }

    # Send log messages to stdout
    console {
        context any info
    }
}

network {
    service {
        address any
        port 3000

        # Uncomment the following to set the `access-address` parameter to the
        # IP address of the Docker host. This will the allow the server to correctly
        # publish the address which applications and other nodes in the cluster to
        # use when addressing this node.
        # access-address <IPADDR>
    }

    heartbeat {

        # mesh is used for environments that do not support multicast
        mode mesh
        port 3002

        # use asinfo -v 'tip:host=<ADDR>;port=3002' to inform cluster of
        # other mesh nodes

        interval 150
        timeout 10
    }

    fabric {
        port 3001
    }

    info {
        port 3003
    }
}

namespace mtm {
    replication-factor 2
    memory-size 1G
    default-ttl 5d # 5 days, use 0 to never expire/evict.

    #   storage-engine memory

    # To use file storage backing, comment out the line above and use the
    # following lines instead.
    storage-engine device {
        file /opt/aerospike/data/mtm.dat
        filesize 4G
        data-in-memory true # Store data in memory in addition to file.
    }
}

How to remove a record completely?

Answer 1

The delete mechanism deletes the index entry to the data, thereby freeing the index space and storage space immediately. However, it does not durably write a tombstone marker record to storage, so deleted records can be restored with a full cold reboot of the cluster or network partition scenarios.

This is from the latest Aerospike Blog Post about the 3.10 release.

2 features available in the Aerospike Enterprise Edition do address this behavior:

1- Fast start (index preserved in shared memory). 2- Durable Deletes (see blog post mentioned above).

You can read more about the behavior you experienced on this thread on the Aerospike Forum.