ServerValue.increment doesn't work properly when Internet goes down

Question

The addition of ServerValue.increment() (Add increment() for atomic field value increments #2437) was a great news as it allows field values ​​to be increased atomically in Firebase RTDB.

I have an application that keeps inventories and this function has been key because it allows updating the inventory regardless of whether the user is offline at times. However, I started to notice that sometimes the function is executed twice, which completely misstates the inventory in the wrong way.

To isolate the problem I decided to do the following test, which shows that ServerValue.Increment() works wrong when the connection goes from Online to Offline:

1. Make a for loop function from 1 to 200:

   for (var i = 1; i <= 200; i++) {
     testBloc.incrementTest(i);
     print('Pos: $i');
   }
   

2. The function incrementTest(i) must increment two variables: position (count from 1 in 1 up to 200) and sum (add 1 + 2 + 3, ..., + 200 which should result in 20,100)

       Future<bool> incrementTest(int value) async {    
        try {    
   
          db.child('test/position')
            .set(ServerValue.increment(1));     
   
          db.child('test/sum')
            .set(ServerValue.increment(value));     
   
        } catch (e) {
          print(e);
        }  
   
   
         return true;
    }
   

Note that db refers to the Firebase instance (FirebaseDatabase.instance.reference())

With this, comes the tests:

Test 1: 100% Online. PASSED

The function works properly, reaching the two variables to the correct result (in the Firebase console):

position: 200

sum: 20100

Test 2: 100% Offline. PASSED

To do this I used a physical device in airplane mode, then I executed the for loop function, and when the function finished executing I deactivated airplane mode and checked the result in the firebase console, which was satisfactory:

position: 200

sum: 20100

Test 3: Start Online and then go to Offline. FAILED

It is a typical operating scenario when the Internet Connection goes down. Even worse when the connections are intermittent, you are traveling on a subway or you are in a low coverage site for which Offline Persistence is a desired feature. To simulate it, what I did was run the for loop function in online mode, and before it finished, I put the physical device in airplane mode. Later I went Online to finish the test and see the results on the Firebase console. The results obtained are incorrect in all cases. Here are some of the results:

As you can see, the Increment was erroneously repeated 10, 18 and 9 times more.

How can I avoid this behavior?

Is there any other way to increment atomically a number in Firebase that works properly online / Offline ?

Answer 1

firebaser here

That's an interesting edge-case in the increment behavior. Between the client and the server neither can be certain whether the increment was executed or not, so it ends up being retried from the client upon the reconnect. This problem can only occur with the increment operation as far as I can tell, as all the other write operations are idempotent except for transactions, but those don't work while offline.

It is possible to ensure each increment happens only once, but it'll take some work:

1. First, add a nonce to write operation that unique identifies this operation. You can use a push key for this, but any other UUID also works fine. Combine this with your original set() call into a single multi-path update call, writing the nonce to a top-level node with a server-side timestamp as its value.
2. Now in your security rules for the top-level location, only allow the write if there is no existing data. This ensures the secondary writes you're seeing get rejected, and since security rules are checked across multi-path updates as a whole, the faulty increment will get rejected too.
3. You'll probably want to periodically clean up the node with nonce keys, based on the timestamp value in there. It won't matter for performance (since you're never searching here outside of during the cleanup), but may help control the storage cost for the nonces.

I haven't used this approach for this specific use-case yet, but have done it for others. If you'd include a client-side retry, the above essentially builds your own multi-path transaction mechanism, which is what I needed it for in the past. But since you don't need that here, it's simpler without that.