Search code examples
google-cloud-datastore

Google Datastore Index Optimization


While designing key-only queries to filter Google Datastore entities, I am generating many composite indexes that are subsets of another index. Is it possible to use the same composite index for queries that filter on a subset of the properties already indexed? For example, if I have the following key-only queries, would it be possible to have less than three indexes?

Query 1: Entities where a = 1, b = 1, c = 1;
Query 2: Entities where a = 1, b = 1;
Query 3: Entities where a = 1;

Here is a sample of the actual query I am working with:

Query<Key> query = Query.newKeyQueryBuilder()
              .setKind("track")
              .setFilter(CompositeFilter.and(PropertyFilter.eq("status", 1), PropertyFilter.eq("bpm", 138), PropertyFilter.eq("artist", "AVB"), PropertyFilter.eq("label", "Armada")))
              .setOrderBy(OrderBy.asc("date"))
              .build();

Solution

  • Datastore can merge smaller indexes together to support larger equality queries, see index merging. Using this feature, a minimal set of indexes for your set of queries would be something like:

    index.yaml

    indexes:
    
    - kind: Albums
      properties:
      - name: artist
      - name: date
    
    - kind: Albums
      properties:
      - name: bpm
      - name: date
    
    - kind: Albums
      properties:
      - name: label
      - name: date
    
    - kind: Albums
      properties:
      - name: status
      - name: date
    

    This supports equality queries on any number of these properties, sorted by date. Note, however, that index merging has a performance trade-off in some cases.