I'm currently exploring using Native Blob Soft Delete to remove data from blobs and have it update in my indexer/index. This is running absolutely fine, and when the indexer runs again, the documents succeeded reflects the changes i've made in soft delete, but unlike when I upload a new document via the indexer, the deletions aren't reflecting in the index?
I've tried calling delete through the REST API like so:
{
"value": [
{
"@search.action": "delete",
"metadata_storage_path": "aHR0cHM6Ly9uYXRpdmVibG9ic3RvcmFnZS5ibG9iLmNvcmUud2luZG93cy5uZXQvYmxvYnRlc3QvUERGcy8yMDEwMDAwNTEzNy5wZGY1",
"content": "Assessment of Prone Positioning of Restrained, Seated Crewmembers in a Post Landing Stable 2 Orion Configuration During the Orion landing and recovery subsystem design review, June 2009, it was noted that the human system and various vehicle systems, the environmental control and life support (ECLSS) and guidance, navigation and control (GN&C) systems for example, are negatively affected by Orion assuming a stable 2 (upside down; Figure A) configuration post landing. The stable 2 configuration is predicted to occur about 50% of the time based on Apollo landing data and modeling of the current capsule. The stable 2 configuration will be countered by an active up-righting system (crew module up-righting system; CMUS). Post landing balloons will deploy and inflate causing the vehicle to assume or maintain the stable 1 (up-right; Figure B) configuration. During the design review it was proposed that the up-righting system could be capable of righting the vehicle within 60 seconds. However, this time limit posed a series of constraints on the design which made it less robust than desired. The landing and recovery subsystem team requested an analysis of Orion vehicle systems as well as the human system with regard to the effect of stable 2 in order to determine if an up-righting response time greater than 60 seconds could be tolerated. The following report focuses on the assessment of the human system in the posture assumed when Orion is in the stable 2 configuration. Stable 2 will place suited, seated, and restrained crewmembers in a prone (facedown), head-up position for a period of time dependent on the functionality of the up-righting systems, ability of the crew to release themselves from the seat and restraints, and/or time to arrival of rescue forces. Given that the Orion seat and restraint system design is not complete and therefore, not available for evaluation, Space Medicine assessed how long a healthy but deconditioned crewmember could stay in this prone, restrained position and the physiological consequences of this posture by researching terrestrial analogs and considered the known physiological alterations and deconditioning experienced by long duration crewmembers.\nDocument ID\n20100005137\nDocument Type\nOther\nAuthors\nBarr, Yael(Texas Univ. Galveston, TX, United States)\nFogarty, Jennifer(NASA Johnson Space Center Houston, TX, United States)\nDate Acquired\nAugust 25, 2013\nPublication Date\nJanuary 1, 2010\nSubject Category\nAerospace Medicine\nReport/Patent Number\nJSC-CN-19414\nDistribution Limits\nPublic\n",
"merged_content": "Assessment of Prone Positioning of Restrained, Seated Crewmembers in a Post Landing Stable 2 Orion Configuration During the Orion landing and recovery subsystem design review, June 2009, it was noted that the human system and various vehicle systems, the environmental control and life support (ECLSS) and guidance, navigation and control (GN&C) systems for example, are negatively affected by Orion assuming a stable 2 (upside down; Figure A) configuration post landing. The stable 2 configuration is predicted to occur about 50% of the time based on Apollo landing data and modeling of the current capsule. The stable 2 configuration will be countered by an active up-righting system (crew module up-righting system; CMUS). Post landing balloons will deploy and inflate causing the vehicle to assume or maintain the stable 1 (up-right; Figure B) configuration. During the design review it was proposed that the up-righting system could be capable of righting the vehicle within 60 seconds. However, this time limit posed a series of constraints on the design which made it less robust than desired. The landing and recovery subsystem team requested an analysis of Orion vehicle systems as well as the human system with regard to the effect of stable 2 in order to determine if an up-righting response time greater than 60 seconds could be tolerated. The following report focuses on the assessment of the human system in the posture assumed when Orion is in the stable 2 configuration. Stable 2 will place suited, seated, and restrained crewmembers in a prone (facedown), head-up position for a period of time dependent on the functionality of the up-righting systems, ability of the crew to release themselves from the seat and restraints, and/or time to arrival of rescue forces. Given that the Orion seat and restraint system design is not complete and therefore, not available for evaluation, Space Medicine assessed how long a healthy but deconditioned crewmember could stay in this prone, restrained position and the physiological consequences of this posture by researching terrestrial analogs and considered the known physiological alterations and deconditioning experienced by long duration crewmembers.\nDocument ID\n20100005137\nDocument Type\nOther\nAuthors\nBarr, Yael(Texas Univ. Galveston, TX, United States)\nFogarty, Jennifer(NASA Johnson Space Center Houston, TX, United States)\nDate Acquired\nAugust 25, 2013\nPublication Date\nJanuary 1, 2010\nSubject Category\nAerospace Medicine\nReport/Patent Number\nJSC-CN-19414\nDistribution Limits\nPublic\n"
}
]
}
This is just using the data direct from the index to make the call in Postman, which is giving me the success code of 200. It has deleted some of the data, so about half the search results have no vanished, the remaining pieces being these two still in the delete call.
After making the call through the API and waiting a short while, the index's documents number does down by one as though it "sees" the deletion of the document, but it's still calling it up when I run the search enquiry, even after waiting for the soft delete time to run out and for the document to become hard deleted.
I'm confident it's the same document, I deliberately picked one which only returned one result in my search, and I ran the document key through a string matching test in python against the one I called in the delete API.
Has anyone encountered this before and got any tips? Or is this as deleted as the document will get without deleting and rebuilding the indexer? I'm exploring this for GDPR, so the fact it still returns a content snippet isn't ideal!
Edit Thought i'd solved this as the documents i'd deleted went yesterday, but this one, specifically the one in the code example, reappeared overnight, which is even weirder because it's now past the soft delete stage in the blob, so there's nowhere for it to be recalling from. Has anyone seen this before?
For anyone else working with this feature, my experience has been that i've just had the occasional buggy file which persisted no matter how many times i've deleted it. Eventually i've had to put this down to being a bug out of the feature still being in preview, as even in the time i've been working with it, updates are being made.