Let's say I have multiple properties that I may or may not want to filter by (see below).
Building up a filter that actually filters by all properties is fairly easy by combining five filter calls and adding one combined index with five properties.
But how do I cover all possible filters where some properties do not require filtering?
One option would be do add indexes for all possible combinations, but that quickly becomes hard to maintain and does not seem like an efficient solution to me.
Another approach I have been thinking about is adding "dummy" filters, for example setting kilometersDriven > -1, however that would exclude any entities that do not have values or have null values for these fields, which is also not what I want.
What's the suggested practice in these cases?
My current workaround is doing "basic" filtering using objectify (e.g. if model is always a required field) and then doing more complex filtering using Java stream filters, however I'm not confident that will be performant enough forever.
@Entity
public class Car {
@Id Long id;
@Index String model;
@Index String color;
@Index Boolean hasAC;
@Index Integer kilometersDriven;
@Index Date purchaseDate;
}
return objectify.get().load().type(Car.class)
.filter("model", model)
.filter("color", color)
.filter("hasAC", hasAC)
.filter("kilometersDriven", kilometersDriven)
.filter("purchaseDate", purchaseDate)
.list();
<datastore-index kind="Car">
<property name="model"/>
<property name="color"/>
<property name="hasAC"/>
<property name="kilometersDriven"/>
<property name="purchaseDate"/>
</datastore-index>
The downside with post processing and filtering additional entities in your app logic is latency and higher number of read ops. But there will be times you can't avoid it.
Refer to the following which talks about zigzag merge join (the main app engine article doesn't seem to be accessible anymore)
Appengine ZigZag Merge Join Algo
This can mitigate the number of indexes still providing flexibility to your filters.