Delete millions of fields in Firebase Realtime Database

Question

I have to delete multiple fields in different locations. I have millions of "messages" with multiple fields like:

messagesId: {
    text: "...",
    datetime: "...",
    unusedField: "...", <-- REMOVE
    ...
}, ...

To save storage space, I want to delete old and not used fields for each message. It means, saves tens of Giga, alias money (hundreds of dollars that we currently pay because in addition to the first GB guaranteed by Firebase).

Problem #1 - Database Peak: deleting a huge amount of data, programmatically, put the Peak to 100%, temporarily blocking it. To solve this, the solution suggested by Firebase support is to use the CLI (firebase database: remove path). Once I've listed millions of lines:

firebase database:remove /root/messages/messageid_1/field --confirm &&
firebase database:remove /root/messages/messageid_2/field --confirm &&
...

even considering a few milliseconds of execution for each line, the overall execution may take an unacceptably long time (days).

Problem #2 - Delete locally and re-upload the DB: another solution suggested, is to download the entire database, remove the json paths and re-upload it. Currently, the entire database weighs 60GB. Is it possible to reload the entire database from the Firebase console? (Given the fact that I would have to suspend any writes in the meantime, to avoid data loss)

Are there any other possible solutions?

Answer 1

The common path for this is:

1. Enable automated backups for your database, and download the JSON from the Storage bucket.
2. Process the JSON locally, and determine the exact path of all nodes to remove.
3. Process the paths in reasonably sized chunks through the API, using multi-location updates.

Removing each chunk of nodes would be something like:

var nodesToRemove = ["/root/messages/messageid_1/field", "/root/messages/messageid_2/field"];

var updates = nodesToRemove.map(function(path) { 
  return { [path]: null };
});
firebase.database().ref().update(updates);