Search code examples
google-app-engineapp-engine-ndbgoogle-search-api

Google Appengine Cursors

I'm using both ndb and search-api queries in my python appengine project.

The only official docs on cursors I can find:

Following things are unclear for me:

  1. What is cursor time-to-live ? Can I expose year-old cursors ?
  2. How would cursor pagination behave in case items are added/removed from original collection? (+ if cursor points to particular record, what happens if this record no longer exists?)
  3. How does query ordering affect above?
  4. Are there any fundamental differences between ndb and search-api cursors?
Solution

  • I'm answering from ndb perspective, I haven't use the search API. All quotes are from your first link.

    For 1 and 3 (as ordering is considered part of the original query from cursors perspective):

    To retrieve additional results from the point of the cursor, the application prepares a similar query with the same entity kind, filters, and sort orders, and passes the cursor to the query's with_cursor() method before performing the retrieval

    So it doesn't really matter how old the cursor is (i.e. how old its query is) since its original query must be restored for the cursor to be obtained.

    For 2:

    Cursors and data updates

    The cursor's position is defined as the location in the result list after the last result returned. A cursor is not a relative position in the list (it's not an offset); it's a marker to which Cloud Datastore can jump when starting an index scan for results. If the results for a query change between uses of a cursor, the query notices only changes that occur in results after the cursor. If a new result appears before the cursor's position for the query, it will not be returned when the results after the cursor are fetched. Similarly, if an entity is no longer a result for a query but had appeared before the cursor, the results that appear after the cursor do not change. If the last result returned is removed from the result set, the cursor still knows how to locate the next result.

    When retrieving query results, you can use both a start cursor and an end cursor to return a continuous group of results from Cloud Datastore. When using a start and end cursor to retrieve the results, you are not guaranteed that the size of the results will be the same as when you generated the cursors. Entities may be added or deleted from Cloud Datastore between the time the cursors are generated and when they are used in a query.

    The Java equivalent page at Limitations of cursors mentions some errors that can be raised for inconsistencies:

    New App Engine releases might change internal implementation details, invalidating cursors that depend on them. If an application attempts to use a cursor that is no longer valid, Cloud Datastore raises an IllegalArgumentException (low-level API), JDOFatalUserException (JDO), or PersistenceException (JPA).

    I suspect Python would be raising some similar errors as well.