Updating entities in ndb while paging with cursors

Question

To make things short, I have to make a script in Second Life communicating with an AppEngine app updating records in an ndb database. Records extracted from the database are sent as a batch (a page) to the LSL script, which updates customers, then asks the web app to mark these customers as updated in the database.

To create the batch I use a query on a (integer) property update_ver==0 and use fetch_page() to produce a cursor to the next batch. This cursor is also sent as urlsafe()-encoded parameter to the LSL script.

To mark the customer as updated, the update_ver is set to some other value like 2, and the entity is updated via put_async(). Then the LSL script fetches the next batch thanks to the cursor sent earlier.

My rather simple question is: in the web app, since the query property update_ver no longer satisfies the filter, is my cursor still valid ? Or do I have to use another strategy ?

Stripping out irrelevant parts (including authentication), my code currently looks like this (Customer is the entity in my database).

class GetCustomers(webapp2.RequestHandler):    # handler that sends batches to the update script in SL
    def get(self):
        cursor=self.request.get("next",default_value=None)
        query=Customer.query(Customer.update_ver==0,ancestor=customerset_key(),projection=[Customer.customer_name,Customer.customer_key]).order(Customer._key)
        if cursor:
            results,cursor,more=query.fetch_page(batchsize,start_cursor=ndb.Cursor(urlsafe=cursor))
        else:
            results,cursor,more=query.fetch_page(batchsize)
        if more:
            self.response.write("more=1\n")
            self.response.write("next={}\n".format(cursor.urlsafe()))
        else:
            self.response.write("more=0\n")
        self.response.write("n={}\n".format(len(results)))
        for c in results:
            self.response.write("c={},{},{}\n".format(c.customer_key,c.customer_name,c.key.urlsafe()))
        self.response.set_status(200)

The handler that updates Customer entities in the database is the following. The c= parameters are urlsafe()-encoded entity keys of the records to update and the nv= parameter is the new version number for their update_ver property.

class UpdateCustomer(webapp2.RequestHandler):
    @ndb.toplevel   # don't exit until all async operations are finished
    def post(self):
        updatever=self.request.get("nv")
        customers=self.request.get_all("c")
        for ckey in customers:
            cust=ndb.Key(urlsafe=ckey).get()
            cust.update_ver=nv   # filter in the query used to produce the cursor was using this property!
            cust.update_date=datetime.datetime.utcnow()
            cust.put_async()
    else:
        self.response.set_status(403)

Will this work as expected ? Thanks for any help !

Answer 1

Your strategy will work and that's the whole point for using these cursors, because they are efficient and you can get the next batch as it was intended regardless of what happened with the previous one.

On a side note you could also optimise your UpdateCustomer and instead of retrieving/saving one by one you can do things in batches using for example the ndb.put_multi_async.