We've installed a cluster of one collection of 350M documents on 3 r3.2xlarge (60GB RAM) Amazon servers. The size of index on each shard is about 1.1TB and maximum storage on Amazon is 1 TB so we add 2 SSD EBS General purpose (1x1TB + 1x500GB) on each instance. Then we create logical volume using LVM of 1.5TB to fit our index.
The response time is about 1 and 3 seconds for simple queries (1 token).
Is the LVM become a bottleneck for our index?
Thanks for help.
Solr is strongly impacted by the storage. Adding a logical volume on top seems like it would have a - possibly - significant overhead.
Have you considered sharding with each shard having it's own SSD?
Also, this kind of question may be better on the User's mailing list as several people may give different views on the situation.