How to mock many urls to return fixture content?

Question

I'm writing some kind of a recursive parser. The simplest form is:

1. Take all links from first link's page body
2. Repeat the first step for each link

So now I want to test it. The problem is I can't figure out the best way to mock all these pages. I use http package and I already have some tests written using httptest package (via httptest.NewServer). But it seems to be no use for my task now. I guess the best way is to use http.Client with custom Transport struct, but it's lots of boilerplate and additional smelly code. Is there a more elegant way to do this?

Answer 1

I have used a custom Transport a couple of times for testing clients. Usually I would create some helper types and functions to cut down on boilerplate code.

Something like this could be a start.

package main

import (
    "bytes"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "os"
)

type roundTripFunc func(*http.Request) (*http.Response, error)

func (r roundTripFunc) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    return r(req)
}

func main() {
    c := &http.Client{
        Transport: roundTripFunc(func(req *http.Request) (resp *http.Response, err error) {
            return &http.Response{
                StatusCode: 200,
                Body:       ioutil.NopCloser(bytes.NewBufferString("test")),
            }, nil
        }),
    }

    r, _ := c.Get("/")
    fmt.Printf("%#v\n", r)
    io.Copy(os.Stdout, r.Body)
}

For example if your testing a JSON API client you could make a round trip helper function takes care of decoding, encoding, headers etc. In your case maybe you could make the round trip function map host header plus URL path into file fixtures paths?