How to change proxy and cookie container per request with the same httpHandler?

Question

I have thousand proxies and thousand custom cookies.

Now I want to send requests with random proxy and using random cookie per request. I have tried to create new httpClientHandler and httpClient each time, but end up using all of my memory of my system.

After reading this article I understand I should reuse the same httpClient. But nor the properties of httpClient and httpClientHandler cannot be modified after the first request.

This instance has already started one or more requests. Properties can only be modified before sending the first request.

So I have used to change the webProxy.Address for changing proxy before sending requests. It works well but failed in the situation of multi-threading. It ends up using wrong proxy for the request. Any other ways to achieve my goal? THANKS.

Addtional Reference: HttpClientHandler / HttpClient Memory Leak

Quote from other users in the above reference:

Thanks for getting to the bottom of this. Unfortunately the HttpClient class does not meet my requirements then - due to the dynamic and unstable nature of public proxies, the objects HAVE to be often re-created. It appears HttpClient is just not a feasible solution for short-living connections -- changing the proxy settings requires re-constructing the HttpClientHandler, and thus the HttpClient. Either way, the objects should be able to live as long or short as needed without leaking; this definitely seems to be a flaw in the HttpClient.

Answer 1

For further investigation and reference for other comments, I believe this can solve my problem.

Instead of reusing the same handler and client, we should recreate it every request for thread safety.

In conclusion, to solve the memory leakage issue for short-lived requests. We can turn off the keep-alive so that the connection will be closed after the responses.

var client = new HttpClient();
client.DefaultRequestHeaders.ConnectionClose = true;

but It turns out this can add roughly extra 35ms (with long tails, i.e amplifying outliers) to each of your HTTP calls preventing you to take advantage of benefits of re-using a socket.

So we can change the keep-alive shorter to due with this instead of switching off keep-alive.

var sp = ServicePointManager.FindServicePoint(new Uri("http://foo.bar/baz/123?a=ab"));
sp.ConnectionLeaseTimeout = 60*1000; // 1 minute

Reference: Singleton HttpClient? Beware of this serious behaviour and how to fix it