Search code examples
goprotocol-buffersgo-modulesprotoc

How to use a package of generated protobuf inside a go module?

I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go@latest).

I have this project structure:

subproj
├── go.mod         (module company.tld/proj/subproj)
├── subproj.go     (entry point : package main)
├── proto          (folder containing .proto files)
├── packageFolder
|   └── file1.go   (package packageFolder)
└── Makefile       (used to generate *.pb.go and build subproj binary)

The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:

syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...

because of different version of messages, few protobuffer files need to be in another "namespace":

option go_package = "company.tld/proj/projpb/other";
package entity.proj.other

In my Makefile, I tried to generate the right *.pb.go at the right place:

# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO=  $(PROTO:.proto=.pb.go)

MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}     
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG=   --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import

.PHONY: clean install proto

## Builds the project
build: proto
    go build ${LDFLAGS} -o ${BINARY}

$(PROTOBUF_GO_PLUGIN):
    go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

$(GRPC_GO_PLUGIN):
    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
    protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<

proto: $(PBGO)

So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import

A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.

Generated Proto objects, that include the other namespace-d objects, have the import path import other "company.tld/proj/projpb/other" (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).

Normal project files need the following import path to reach the Generated Proto objects :
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
which seems odd and not the proper way to do.

→ With --go_opt=module=company.tld/proj

A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.

Generated Proto objects, that include the other namespace-d objects, still have the import path import other "company.tld/proj/projpb/other" (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).

The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb "company.tld/proj/subproj/projpb".

Finally, I tried

  • using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other "./projpb/other" in generated protobuffer object)
  • to use the replace instruction in the go.mod file like this :
replace (
    company.tld/proj/projpb => ./projpb
    company.tld/proj/projpb/other => ./projpb/other
)

(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)

Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».

[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:

  • generated protos use the full path, specified with the go_package option, to relate to one another.
  • As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.

What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?

Solution

  • After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...

    final proto «header» (respects the full puth in the go_package option)

    syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// or for subpackages...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";

    my Makefile (creates a go.mod file for the generated proto files)

    
# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO=  $(PROTO:.proto=.pb.go)

DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}

PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod

.PHONY: clean install proto gomod

build: proto gomod
    go build ${LDFLAGS} -o ${BINARY}

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
    ${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<

proto: $(PBGO)

gomod: ${PROTO_GOMOD_FILE}

${PROTO_GOMOD_FILE}:
    cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..

    my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)

    module company.tld/proj/subproj

go 1.16

require (
    // ...
    company.tld/proj/projpb v0.0.0
)

replace company.tld/proj/projpb v0.0.0 => ./projpb

    Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
    Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
    And the import statement to use generated protos are working fine in the main project.

    I hoped there was a simpler and more prettier solution, but alas...

    Sorry for the any and thanks to those who gave it a thought !