Search code examples
nginxproxyvps

Can I put Nginx as proxy in front of other proxy pool?


I've written a backend service which periodically connects to websites and collects some data. This service supports the usage of proxy servers for sending it's requests.

Right now this service has a predefined (hardcoded) pool of proxies. For each request it selects a random proxy from the pool and uses it. The service is scalable and having multiple instances running at the same time.

It looks like this: enter image description here

The proxies I am using do not have limitation on amount of the concurrent connections. However, if multiple connections are open with same proxy server, they should come from the same IP address. This is not a problem while all service instances are running on the same machine, but I am thinking about buying an extra VPS to run more instances there.

Updated architecture will look like this: enter image description here

I'm thinking if there is a possibility to put Nginx or a similar tool in front of my proxy pool, which will play the role of proxy itself. Something like this: enter image description here

Could you please help me to understand, what would be the most efficient way to implement this use case?


Solution

  • I did my very-very best trying to apply the solution from @VonC's answer, played with headers configuration a lot, but unfortunately it never worked for me, the proxy server was always returning an error with 400 status code: Bad Request.

    The solution which actually worked for me was as simple as this:

    stream {
        upstream proxy_pool {
            server <proxy1_ip>:<port>;
            server <proxy2_ip>:<port>;
            random;
        }
    
        server {
            listen 3333;
            proxy_pass proxy_pool;
        }
    }
    

    So instead of using a http module I used stream module, which is actually installed with Nginx server by default (didn't need to build it separately). This way I also don't need to add any additional header configurations.

    Using this approach I also don't need to store the credentials for proxies on my Nginx server. When service is using Nginx server as proxy, it passes the credentials as if they were meant for my Nginx proxy, and Nginx is redirecting the request with given credentials to the actual proxy.

    With this set up the whole flow looks like this:

    My Service ---login/password---> Nginx Proxy ---same (redirected) login/password---> Random proxy from proxy_pool ---> Website

    As a result, the proxy servers "think" that all requests are originating from the IP of my Nginx server, even if they were originally sent from different instances of my VPSs. This way I was able to walk around of the proxy limitations.