Delete command not removing records from Aerospike

Question

I'm using the following method from Java Aerospike client in order to delete records/bins:

 def truncate(startTime: Long, durableDelete: Boolean): List[AtomicInteger] = {
    logger.info(s"truncate($startTime) Triggered")

    val calendar = new GregorianCalendar()
    calendar.setTimeInMillis(startTime)

    // Define Scan and Write Policies
    val scanPolicy = new ScanPolicy()
    scanPolicy.filterExp = Exp.build(Exp.le(Exp.lastUpdate(), Exp.`val`(calendar)))
    val writePolicy = client.writePolicyDefault
    writePolicy.durableDelete = durableDelete

    // Scan all records such as LUT <= startTime
    for (recoverBins <- config.binsToRecover) yield {
      val recordCount = new AtomicInteger(0)
      client.scanAll(scanPolicy, recoverBins.namespace, recoverBins.set, new ScanCallback() {
        override def scanCallback(key: Key, record: Record): Unit = {
          recoverBins.specificBins match {
            // multi-bin scenario
            case Some(specificBins) =>
              specificBins foreach (bin => client.put(writePolicy, key, Bin.asNull(bin)))
              logger.debug(s"Bins $specificBins of record: $record with key: $key are set to NULL")

            // single-bin scenario
            case None =>
              client.delete(writePolicy, key)
              logger.debug(s"Record: $record with key: $key DELETED")
          }
          recordCount.incrementAndGet()
        }
      })

duruableDelete is set to true

The problem is that when I'm removing bins (Bin.asNull) i can see the results immediately but when checking the "deleted" key i can still see them (aql> select * from ns.set where PK = <ShouldBeDeleted>

any ideas why? what I'm doing wrong?

---

Another question: How does duruableDelete is preventing "Zombie records"? what happening behind the scene?

Thanks!

Answer 1

Zombie Record: A record that was earlier removed from the cluster but comes back alive in the cluster at a later time. There are various scenarios under which this can happen. Durably deleting records prevents all but one of these scenarios. Durable deletes do this by saving a tombstone in the cluster. Example: You have a record on master and replica. You shutdown master node, then delete record in the cluster non-durably. When you restart the master, the record will be resurrected. If you durably deleted, the cluster will have a tombstone (assuming you restarted master within tombstone life period - 1 day default) and when incoming master compares record metadata upon joining the cluster, it will not resurrect the its copy of the record.