JDO Query with cursors on GAE returns inconsistent results

Question

I am running GWT RPC calls to a GAE server, querying for article objects that are stored in the datastore with JDO, and I am paginating the results by using cursors.

I send an initial RPC call to start the pagination with a "range" of 10 results. I store the query cursor in the memcache, and retrieve it when the user requests for the next page of 10 results. The code that implements this is shown below.

The range is always the same, 10 results. However, some subsequent RPC calls return 2 results, or 12 results. It is very inconsistent. The calls also sometimes return duplicate results.

I have read this Google developers documentation: https://developers.google.com/appengine/docs/java/datastore/queries#Java_Limitations_of_cursors. It mentions that: "Cursors don't always work as expected with a query that uses an inequality filter or a sort order on a property with multiple values. The de-duplication logic for such multiple-valued properties does not persist between retrievals, possibly causing the same result to be returned more than once."

As you can see in the code, I am sorting on a "date" property. This property only has one value.

Can you let me see what I am doing wrong here. Thanks.

This is the code that executes the RPC call on the GAE server:

import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.List;
import java.util.logging.Logger;

import javax.jdo.PersistenceManager;
import javax.jdo.Query;
import javax.servlet.http.HttpSession;

import com.google.appengine.api.datastore.Cursor;
import com.google.appengine.datanucleus.query.JDOCursorHelper;
import com.google.gwt.user.server.rpc.RemoteServiceServlet;

//...

private void getTagArticles(String tag, int range, boolean start) {

    PersistenceManager pm = PMF.getNonTxnPm();
    ArticleStreamItemSummaryDTO aDTO = null;
    ArticleStreamItem aDetached = null;
    summaryList = new ArrayList<ArticleStreamItemSummaryDTO>();
    String cursorString = null;
    session = getThreadLocalRequest().getSession();
    UserAccount currentUser = LoginHelper.getLoggedInUser(session, pm);
    String cursorID = currentUser.getId().toString() + tag;

    if (start) { // The start or restart of the query
        CacheSupport.cacheDelete(String.class.getName(), cursorID);
    }

    Object o = CacheSupport.cacheGet(String.class.getName(), cursorID);
    if (o != null && o instanceof String) {
        cursorString = (String) o;
    }

    Query q = null;
    try {
        q = pm.newQuery(ArticleStreamItem.class);

        if (cursorString != null) {
            Cursor cursor = Cursor.fromWebSafeString(cursorString);
            Map<String, Object> extensionMap = new HashMap<String, Object>();
            extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);
            q.setExtensions(extensionMap);
        }   
        q.setFilter("tag == tagParam");
        q.declareParameters("String tagParam");
        q.setOrdering("date desc");
        q.setRange(0, range);

        @SuppressWarnings("unchecked")
        List<ArticleStreamItem> articleStreamList = (List<ArticleStreamItem>) q.execute(tag);

        if (articleStreamList.iterator().hasNext()) {
            Cursor cursor = JDOCursorHelper.getCursor(articleStreamList);
            cursorString = cursor.toWebSafeString();
            CacheSupport.cacheDelete(String.class.getName(), cursorID);
            CacheSupport.cachePutExp(String.class.getName(), cursorID, cursorString, CACHE_EXPIR);
            for (ArticleStreamItem a : articleStreamList) {
                aDetached = pm.detachCopy(a);
                aDTO = aDetached.buildSummaryItem();
                summaryList.add(aDTO);
            }
        }
    }
    catch (Exception e) {
        // e.printStackTrace();
        logger.warning(e.getMessage());
    }
    finally {
        q.closeAll();
        pm.close();
    }
}

Answer 1

The code snippet that I provided in the question above actually works well. The problem arose from the client side. The RPC calls were sometimes being made a few milliseconds from each other, and that was creating the inconsistent behavior that I was seeing in the results returned.

I changed the client side code to make a single RPC call every 5 seconds, and that fixed it.