Golang ReverseProxy per host

Question

I am trying to implement a Reverse Proxy in Go that proxies traffic to different hosts based on some tenant embedded in the URL. The implementation looks like this:

type Offloader struct {
    tenantHostMap   map[string]string                   // Map a tenant to its host:port
    tenantProxyMap  map[string](*httputil.ReverseProxy) // Map a tenant to its reverse proxy
}

func (o *Offloader) OnCreate() {

    // Tenants Map
    o.tenantHostMap = make(map[string]string)
    o.tenantProxyMap = make(map[string]*httputil.ReverseProxy)
    o.PopulateTenantHostMap()

    // Rx
    http.HandleFunc("/", o.ServeHTTP)
    go http.ListenAndServe(":5555", nil)

}

// ServeHTTP is the callback that is called each time a Http Request is received.
func (o *Offloader) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    incomingUrl := req.URL.RequestURI()
    tenant := o.GetTenantFromUrl(incomingUrl)

    if proxy, ok := o.tenantProxyMap[tenant]; ok {
        proxy.ServeHTTP(w, req)
    }

    if remoteHostAddr, ok := o.tenantHostMap[tenant]; ok {
        remoteUrl, err := url.Parse(fmt.Sprintf("http://%s", remoteHostAddr))
        if err != nil {
            return
        }
        proxy := httputil.NewSingleHostReverseProxy(remoteUrl)
        o.tenantProxyMap[tenant] = proxy
        proxy.ServeHTTP(w, req) // non blocking

    } else {
        panic("Unknown Tenant")
    }
}

When receiving a new HTTP request, I get the tenant from the URL. If this is the first time I am seeing this tenant I create a new ReverseProxy, otherwise I try to use the one I created before and stored in the tenantProxyMap.

When I test this, I get the following error:

2022/04/05 12:31:01 http: proxy error: readfrom tcp ****: http: invalid Read on closed Body
2022/04/05 12:31:01 http: superfluous response.WriteHeader call from net/http/httputil.(*ReverseProxy).defaultErrorHandler (reverseproxy.go:190)

If I create a new Reverse Proxy for each request rather than reusing the same proxy, the error doesn't happen.

I thought the proxy is per host and not per request (as the name suggests), so I am wondering why this error happens?

I know I need to protect the maps from concurrent reads/writes however that is irrelevant at the moment.

Thanks,

Answer 1

The problem is that in the scenario where a previous proxy already existed, you first pass the request on to that - and then still recreate the proxy, and again pass the request. In other words: you are making two proxied requests for each incoming request, when the tentantProxyMap is already populated for that tenant.

The ReverseProxy implementation closes the req.Body, so the second time you pass the request on to the proxy, it attempts reading from an already closed body. You're seeing the http: invalid Read on closed Body error as a result.

What you should try is to return after proxying the request, e.g. by adding a return:

if proxy, ok := o.tenantProxyMap[tenant]; ok {
    proxy.ServeHTTP(w, req)
    return
}