Does anyone know of a benchmark to evaluate the reasoning performance of a triple store? I am using Stardog which uses Pellet as its reasoner engine and looking for a benchmark to assess the reasoning performance.
If there is no then any suggestion regarding how to do the evaluation is appreciated.
Thanks.
LUBM is probably the most standard benchmark. The LDBC is doing some work on a reasoning benchmark, but at last look, it was not well thought through and still in its early stages.
As with all benchmarks, be careful in overly relying on benchmark results. All that really tells you is that a particular system is good at that benchmark. It might be indicative of the performance, generally, and it might not. Unless you're trying to build something on the benchmark, you should be considering how you can create a benchmark with your own data and queries to be reflective of your own, actual use cases.
You should also be mindful of the characteristics of the different reasoning implementations, particularly the implications of materialization vs query-rewriting, and take that into account when interpreting your results.