My website is growing. I need to partition feature sets to different group of servers instead of having all features run on a single server copy. I have 8 identical servers in a Windows Network Load Balancing setup.
The problem is: I need to keep the URL space the same. What would be a workable solution? I don't want to start new domains for the feature sets. I am thinking some kind of reverse proxy with some URL based rewriting/routing capabilities(?!?). Any recommendations in terms of software or hardware? This is going to replace the WNLB setup as it does not have the capabilities I need.
There are indeed several solutions to implement load balancing:
I understood 1) and 2) are not an option here so... if you have money and really high performance needs, go for 4). Else, go for 3).
For Proxy Round-Robin, again, several solutions are possible: Apache mod_rewrite, Apache mod_proxy, Squid (and surely many others I don't know).
For "dumb" load balacing, there is an example in Apache mod_rewrite's URL Rewriting Guide (see Proxy Throughput Round-Robin section).
Apache mod_proxy can act as proxy to connect clients to the internet but is usually used as reverse proxy to redirect an url to another server. It has no cache functionality (but can be used with mod_cache, and mod_rewrite...).
Squid is a proxy cache and is usually used to connect clients to the internet. But it can also be used as reverse proxy and be configured to cache the requests and to accelerate content delivery.
As you can see, choosing one of them depends on what and how you want to proxy. In your case, I would consider running Apache mod_proxy or Squid if you want caching on Linux (if it is an option).
Regarding the hardware, I'm not a specialist but I think a "small" to "medium" dedicated server should be enough. Just don't forget all the requests will go through this machine so its sizing highly depends on your traffic which seems decent. This might require some digging with real life data.