replace retool with tools.go for multi developer and CI environments using go modules

I'd like to replace retool with go modules tools.go "tools as dependencies". However I'm struggling to understand how this works when my developers and CI env all use different operating systems.

I want to make sure each environment uses the exact same version of tools.

For a concrete example, my app requires the protoc compiler to generate go code via github.com/golang/protobuf/protoc-gen-go. I have 3 OS, all needing to execute protoc with the protoc-gen-go plugin/generator:

Ryan: uses MacOS
Joe: Linux (Ubuntu)
CI: Linux (CentOS)

I currently use retool to make sure ALL environments are locked in on the same version of tools (protoc-gen-go in this ex):

retool do build/bin/protoc -Ibuild/protoc/include -I. rpc/platform/platform.proto --go_out=.

My new go modules / "tools as dependencies" setup

tools.go:

// +build tools

package tools

import (
    _ "github.com/golang/protobuf/protoc-gen-go"
)

Set the path that go install will use:

export GOBIN=$PWD/bin

Install:

go install github.com/golang/protobuf/protoc-gen-go

If Ryan runs the go install .. a bin/protoc-gen-go MacOS executable is created.

Questions:

At this point, why isprotoc-gen-go tool version (or git hash) NOT listed in go.mod?
When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?
How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?

Solution

I was able to accomplish vendored tools build for protoc (and plugins like Twirp) following the Go Modules tools guidelines, plus a little Makefile-Fu for the protoc binary.

A complete working example can be found in the Aspiration Labs pyggpot repo. Following are the essential details. Worth noting: getting the import path right for some of the tools was very fiddly, but ultimately, successful.

For protoc itself, I vendor the binary release in the Makefile and set it up into a tools/bin dir:

TOOLS_DIR := ./tools
TOOLS_BIN := $(TOOLS_DIR)/bin

# protoc
PROTOC_VERSION := 3.7.1
PROTOC_PLATFORM := osx-x86_64
PROTOC_RELEASES_PATH := https://github.com/protocolbuffers/protobuf/releases/download
PROTOC_ZIP := protoc-$(PROTOC_VERSION)-$(PROTOC_PLATFORM).zip
PROTOC_DOWNLOAD := $(PROTOC_RELEASES_PATH)/v$(PROTOC_VERSION)/$(PROTOC_ZIP)
PROTOC := $(TOOLS_BIN)/protoc

# protoc
$(PROTOC): $(TOOLS_DIR)/$(PROTOC_ZIP)
    unzip -o -d "$(TOOLS_DIR)" $< && touch $@  # avoid Prerequisite is newer than target `tools/bin/protoc'.

$(TOOLS_DIR)/$(PROTOC_ZIP):
    curl --location $(PROTOC_DOWNLOAD) --output $@

The PROTOC_PLATFORM string can be automated with something like OS detecting makefile. The version of that we use is at https://github.com/aspiration-labs/pyggpot/blob/master/build/makefiles/osvars.mk.

On to building the go tools. Create a tools.go something like

// +build tools

package tools

import (
    // protocol buffer compiler plugins
    _ "github.com/golang/protobuf/protoc-gen-go"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp_python"
    _ "github.com/thechriswalker/protoc-gen-twirp_js"
)

Note: the // +build tools tag will keep go build from over-building tools imports in your final build.

Finally, some make code to build your go tools:

# go installed tools.go
GO_TOOLS := github.com/golang/protobuf/protoc-gen-go \
            github.com/twitchtv/twirp/protoc-gen-twirp \
            github.com/twitchtv/twirp/protoc-gen-twirp_python \
            github.com/thechriswalker/protoc-gen-twirp_js \

# tools
GO_TOOLS_BIN := $(addprefix $(TOOLS_BIN), $(notdir $(GO_TOOLS)))
GO_TOOLS_VENDOR := $(addprefix vendor/, $(GO_TOOLS))

setup_tools: $(GO_TOOLS_BIN)

$(GO_TOOLS_BIN): $(GO_TOOLS_VENDOR)
    GOBIN="$(PWD)/$(TOOLS_BIN)" go install -mod=vendor $(GO_TOOLS)

And finally, a make setup target to run go mod vendor and process the targets above.

setup: setup_vendor $(TOOLS_DIR) $(PROTOC) setup_tools

# vendor
setup_vendor:
    go mod vendor

$(TOOLS_DIR):
    mkdir -v -p $@