Search code examples
pythongoogle-app-enginegoogle-cloud-datastore

ndb datastore query cursor and indexing issues


We are using ndb Datastore, python, standard google app engine. We would like to use Query Cursor. But for this to work according to here and here, it looks like we need to implement datastore_model.query().order(-datastore_model.key).

For example in our query we have

teacher_model_query     = teacher_model.query(ndb.AND(
                ndb.GenericProperty('signinout_time') >= signinout_time_start, 
                ndb.GenericProperty('signinout_time') <= signinout_time_end))

teacher_query_forward = teacher_query.order(ndb.GenericProperty('signinout_time')).order(teacher__model.key)
teacher_query_reverse = teacher_query.order(-ndb.GenericProperty('signinout_time')).order(- 
 teacher__model.key)

Unfortunately this means that we have to create a new index for this

- kind: teacher_model
  properties:
  - name: signinout_time
    direction: desc
  - name: __key__
    direction: desc

This eats into the 200 indexes limit per project. Can you please confirm that we need to have order(-datastore.model.key) for query cursor to work in reverse direction? How can we execute Query cursor without the need to create additional indexes?


Solution

  • You need to create such index only if your app needs the ability to move backwards in the query result pages. Which may be desired, for example, if you offer a page navigation functionality and you allow such navigation to a previous page.

    But such functionality is usually not needed if you use cursors simply to split processing loads into smaller batches - in such cases you only navigate in one direction.