How to reuse an http.Client instance with a dynamic proxy URL in Go

Question

I'm building an HTTP client in Go and I need to make calls to an endpoint via a proxy server. The proxy address is not fixed, so I'm currently creating a new http.Client instance every time I need to make a call. This isn't very efficient, as it involves creating a new transport and client object every time.

Is there a way to create the http.Client instance only once and then update the proxy address before making a call? I'd like to avoid creating a new client instance every time I need to make a request.

One more thing - There will be a fixed number of proxy servers (<5) and hence fixed number of URLs but I don't know the address in the beginning of those servers, and I will get those addresses dynamically.

Because of this condition, I am thinking if I could create multiple client - one for each server and reuse them based on the URL I get. But I would still prefer a solution where I could create only one client and update proxy URL somehow.

Any suggestions would be very helpful.

Here's my current code for creating the client:

proxyURL, err := url.Parse("http://proxy_url:proxy_port")
if err != nil {
    fmt.Println(err)
    return
}

transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
}
client := &http.Client{
    Transport: transport,
}

Thanks!

Answer 1

You are already using the Proxy field of the Transport, but there is no reason to use http.ProxyURL (which returns a static value). Write your own lookup function instead:

package main

import (
    "net/http"
    "net/url"
)

func main() {
    c := &http.Client{
        Transport: &http.Transport{
            Proxy: lookupProxy,
        },
    }

    // use (and re-use) c for any request
}

func lookupProxy(r *http.Request) (*url.URL, error) {
    // TODO: map r.URL to the appropriate proxy
    panic("unimplemented")
}

If the decision which proxy to use depends on things not available in http.Request, you make it before calling Client.Do and pass the proxy URL via the request context:

package main

import (
    "context"
    "net/http"
    "net/url"
)

func main() {
    c := &http.Client{
        Transport: &http.Transport{
            Proxy: ProxyFromContext,
        },
    }

    req, err := http.NewRequest("GET", "http://example.com", nil)

    req = WithProxy(req, &url.URL{
        // ...
    })

    res, err := c.Do(req)


    // use (and re-use) c for any request
}

type ctxKey int

const (
    ctxKeyProxyURL ctxKey = iota
)

func WithProxy(r *http.Request, u *url.URL) *http.Request {
    ctx := context.WithValue(r.Context(), ctxKeyProxyURL, u)

    return r.WithContext(ctx)
}

func ProxyFromContext(r *http.Request) (*url.URL, error) {
    u, _ := r.Context().Value(ctxKeyProxyURL).(*url.URL)
    return u, nil
}