Arduino ESP edit big JSON file - wear leveling

Question

I have a "big" JSON file about 1000 objects.
Using https://arduinojson.org/ to serialize / deserialize.

void ConfigFile_Save_Variable(String VarName, String VarValue) {
  Serial.print("ConfigFile_Save_Variable: ");
  Serial.print(VarName); Serial.print("="); Serial.print(VarValue);

  File configFile = SPIFFS.open("/config.json", "r");
  if (!configFile) {
    Serial.println("- failed to open config file for writing");
    return;
  }

  DynamicJsonDocument doc(2048);
  deserializeJson(doc, configFile);
  serializeJson(doc, Serial);
  configFile.close();
  //
  doc[VarName] = VarValue;
  //
  configFile = SPIFFS.open("/config.json", "w");
  serializeJson(doc, configFile);
  serializeJson(doc, Serial);
  configFile.close();
  Serial.println("");
  Serial.println(" - config.json saved - OK.");

If I want to edit only 1 object and save the JSON file, the whole file is written again.
This is bad for wear leveling, as it write all data again tho only 1 object has been changed.

So I'm looking for a way to only write the change not the whole file. Tried looking everywhere without luck.

Answer 1

The best way to handle this is to break up the JSON - write each top level object to its own file. You already have the labels for the objects, use those as filenames prefixed with /config/ or /c/ if you need more space for the label name.

Let's look at why what you're asking for from SPIFFs doesn't work.

JSON is serialized for storage as text. For example, the object

{
  label_0: {
    label_a: "small string",
    label_b: 1
    },
  label_1: {
    label_c: "another string",
    label_d: "more string"
  }
}

is serialized as

{"label_0":{"label_a":"small string","label_b":1},"label_1":{"label_c":"another string","label_d":"more string"}}

SPIFFS splits stored files into pages of fixed size. It has to write an entire page to flash each time - it can't just write the changed bytes; that's not how flash storage works.

Suppose label_a becomes shorter - just "s". Then everything after label_a would have to be shifted down in the file. For SPIFFS to do what you're asking, it would have to write a page and remember that only part of the page was in use.

If the value of label_a becomes larger - "this is a much longer string" then everything would have to be shifted up in the file. SPIFFS would have to allocate a new page to store the overflow and remember that only part of it was in use.

Keep in mind that the only way SPIFFS can remember which blocks are in use is by writing that information to flash as well.

That's a lot to ask of a simple filesystem.

Let's also get some perspective on the situation. How big is the file, and how often is it written? If you have 1000 objects and each object takes, say, 200 bytes to store, your JSON would be about 20,000 bytes. An average ESP8266 or ESP32 has 4MB of flash, with maybe 1.5MB available for SPIFFS. You can store 20,000 bytes 75 times in 1.5MB - so you probably have a factor of around 75x or a bit less for wear leveling.

The flash in most ESP8266's and ESP32's is rated for 100,000 writes (in practice it would usually last much longer). So with wear leveling you could probably write a 20,000 byte file 750,000 times without expecting to see errors.

Are you actually likely to write this file even 1000 or 10,000 times?

If you are - this seems like an odd scenario for a configuration file, and you should rethink how you're handling this file and what it's doing.

If not then this isn't actually a problem.