Search code examples
c#algorithmsearchindexingin-memory-database

Efficient design to search for objects by multiple parameters with range


I have a set of objects of the same type in memory and each has multiple immutable int properties (but not only them).

I need to find an object there (or multiple) whose properties are in small range near specified values. E.g. a == 5+-1 && b == 21+-2 && c == 9 && any d.

What's the best way to store objects so I can efficiently retrieve them like this?

I thought about making SortedList for each property and using BinarySearch but I have a lot of properties so I would like to have a more generic way instead of so many SortedLists.

It's important that the set itself is not immutable: I need an ability to add/remove items.

Is there something like memory db for objects (not just data)?


Solution

  • Just to expand on @j_random_hacker's answer a bit: The usual approach to 'estimates of the selectivity' is to build a histogram for the index. But, you might already intuitively know which criteria is going to yield the smallest initial result set out of "a == 5+-1 && b == 21+-2 && c == 9". Most likely it is "c == 9" unless there's an exceptionally high number of duplicate values and small universe of potential values for 'c'.

    So, a simple analysis of the predicates would be an easy starting point. Equality conditions are highly likely to be the most selective (exhibit the highest selectivity).

    From that point, RDBMS' will conduct a sequential scan of the records in the result set to filter for the remaining predicates. That's probably your best approach, too.

    Or, there's any number of in-memory, small footprint SQL-capable DBMS that will do the heavy lifting for you (eXtremeDB, SQLite, RDM,... google is your friend) and/or that have lower-level interfaces that won't do all the work for you (still, most) but also won't impose SQL on you.