Creating a wrapper around go-retry RetryFunc to accept any API definition

Question

I am using the https://pkg.go.dev/github.com/sethvargo/go-retry package to make retry attempts on the GitHub API server from my application. The package is the simplest of the lot, that assists in making retry attempts of a web API call.

The issue that is bugging me is the RetryFunc in https://pkg.go.dev/github.com/sethvargo/go-retry#RetryFunc takes a closure of the form

type RetryFunc func(ctx context.Context) error

I have a lot of GitHub APIs wrapped around this retry logic, with varying number of arguments.

Sample code below

func retryDecision(resp *github.Response) error {
    if resp.StatusCode == 0 || (resp.StatusCode >= 500 && resp.StatusCode != http.StatusNotImplemented) {
        return retry.RetryableError(fmt.Errorf("unexpected HTTP status %s", resp.Status))
    }
    return nil
}

func retryWithExpBackoff(ctx context.Context, f func(ctx context.Context) error) error {
    b := retry.NewExponential(1 * time.Second)
    b = retry.WithMaxRetries(5, b)
    b = retry.WithMaxDuration(5*time.Second, b)
    return retry.Do(ctx, b, f)
}

func NewGitHubClient(ctx context.Context, token, repo string) (*github.Client, error) {
    r := strings.Split(repo, "/")
    client := github.NewClient(nil).WithAuthToken(token)

    if err := retryWithExpBackoff(ctx, func(ctx context.Context) error {
        _, resp, err := client.Repositories.Get(ctx, r[0], r[1])
        if retryErr := retryDecision(resp); retryErr != nil {
            return retryErr
        }
        return err
    }); err != nil {
        return nil, err
    }

    return client, nil
}

I end up nesting calls like below for each of the API that I'm implementing with retry, i.e. this part of code is repeating in each of the APIs I plan to implement

    if err := retryWithExpBackoff(ctx, func(ctx context.Context) error {
        // <-------- API that needs to run by retry --------->
        //_, resp, err := client.Repositories.Get(ctx, r[0], r[1])
        if retryErr := retryDecision(resp); retryErr != nil {
            return retryErr
        }
        return err
    }); err != nil {
        return nil, err
    }

How can I re-write this to make the retryWithExpBackoff take any API that I plan to implement, but adopt a retry mechanism?

The GitHub API referenced in the code is from https://pkg.go.dev/github.com/google/go-github/v61/github

Answer 1

I managed to find a way to re-write the retry logic this way. But would still appreciate other ways.

type GitHubAPIRequest func(ctx context.Context) (*github.Response, error)

func retryWithExpBackoff(ctx context.Context, request GitHubAPIRequest) error {
    b := retry.NewExponential(1 * time.Second)
    b = retry.WithMaxRetries(5, b)
    b = retry.WithMaxDuration(5*time.Second, b)
    return retry.Do(ctx, b, func(ctx context.Context) error {
        response, err := request(ctx)
        if retryErr := retryDecision(response); retryErr != nil {
            return retryErr
        }
        return err
    })
}

func NewGitHubClient(ctx context.Context, token, repo string) (*github.Client, error) {
    r := strings.Split(repo, "/")
    client := github.NewClient(nil).WithAuthToken(token)

    request := func(ctx context.Context) (*github.Response, error) {
        _, resp, err := client.Repositories.Get(ctx, r[0], r[1])
        return resp, err
    }

    if err := retryWithExpBackoff(ctx, request); err != nil {
        return nil, err
    }

    return client, nil
}

Though this works, it limits my APIs to the signature of GitHubAPIRequest and in case if I need to access the response from the GitHub API other than just the error, I might have to tweak it a bit more.

---

I managed to work-around this problem by using a retryable http client within the github client, so that the application logic doesn't need to worry about doing retries manually.

// NewGitHubClient returns an instance of a GitHub client to interact with a
// remote repository with an OAuth token as an argument.
func NewGitHubClient(token string) *github.Client {
    // The benefit of this approach of injecting a retryable http client to the
    // GH client API is that is allows us to push the retry logic down the stack,
    // and the application code itself does not need to care or implement how
    // retries are being done.
    retryableClient := retryablehttp.NewClient()
    client := github.NewClient(retryableClient.StandardClient()).WithAuthToken(token)

    return client
}