I am trying to test SOLR performance for SOLR 5.2 (which I am aiming to switch to) vs SOLR 3.6 (currently used in prod), I am using Jmeter to perform the test, with 6 queries in the test plan , all are simple queries (either querying for a fixed field value, ex : some_field:"fixed_value", or a simple facet) .... the problem is that I am expecting the performance to be better after the enhancements done through the versions, but always it gives worse results (worse average response time), so I am asking :
1) if I am doing something wrong in the tests / test criteria that give unaccurate results,
2)or may be because the queries use fixed values, so reading from the cache or soemthing like that
3)not appropriate to test on my local (non server machine)
4)need to tune jmeter
5)this is normal
environment details :
win7 - 8GB RAM - i7
apache tomcat 7/8 - jetty (I tried all the three for both 3.6 and 5.2)
java 7/8 (I tried both for both 3.6 and 5.2)
30M documents in each solr core
the core is indexed using the same version of solr it's imported in
*test details (in jmeter):
60 users
20s ramp up period
throughput 20 requests/second (this is the expected hit rate, so I tuned a delay to reach it)
6 queries per user (running infinetly)
Your question is about performance of solr 5.2 with only 6 queries and simple facet search. No range queries, not complex queries and not updates (which would clear the caches).
We recently also switched from 3.X version to 5.X. For your simple queries and simple facets we also had a huge (factor 0.8) performance regression. But overall the search application was a lot (factor 1.2) faster.
Since 3.6 a lot of work in lucene was for near realtime search (NRS). Mainly the lucene-folk reduced the memory impact and go from IndexReader to LeafReader (=reader per segment).
Other great improvement is that lucene now knows more about the kind of tokens in the index (BytesRef instead of Strings) and can use a Automaton to go through the index (instead of only be good to skip from one token to an other like for prefix search).
The fast access to all document per term, was the focus of lucene form the beginnings and hasn't got to improve.
For sure the simple faceting in solr 3.X was the peek of performance in ms because mostly the facet values were all in main memory. So very fast but expensive and only fast without updates of index.
Possible you can switch the codec to get a better "search for fixed field value" performance. Most possible you could benefit from the new DocValues for your facet search if you switch to Version 5.4 (because it speed-up faceting on doc values fields).
Be aware that the solr-folk waiting for a proof that faceting is now really slower. If you can proof this also with 5.4 write something in issue SOLR-8096: Major faceting performance regressions
Update about faceting:
SOLR-8096 is still open, but solr 5.5 seems to be fast for faceting again via SOLR-8466: Uninverted field faceting is re-enabled, for higher performance on rarely changing indices