Search code examples
mavennugetprotocol-buffersbuf

openapiv2 imports causes compilation error in generated code

I am using buf to generate grpc server and client code for several languages (go, python, js, java, c#), while using grpc-ecosystem/plugins/openapiv2 plugin to generate swagger documentation from the same proto files.

In some files I'm using custom option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_tag) = {description: "Manage datasets and examples used for training."}; to add additional metadata to the documentation. This requires me to import annotations.proto from grpc-gateway project which causes the imports to also appear in generated source files. Now languages like go for example can handle this by using import for side effects

import (
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options"
_ "google.golang.org/genproto/googleapis/api/annotations"
)

but in java and c# there are some lines that are being generated which look like this

registry.add(com.google.api.AnnotationsProto.http);
registry.add(grpc.gateway.protoc_gen_openapiv2.options.Annotations.openapiv2Tag);

which causes compilation errors, because package grpc.gateway... does not exist (I was able to import the googleapis package via Maven and nuget). When I remove the options from .proto files there are no issues and I can compile the source files to a package for distribution. Is there any way to exclude these imports from generated code?

I have tried separating the documentation to its own files, but it's impossible to do with metadata which are part of Service or Message definitions as I'm getting duplicate definition errors.

Solution

  • Since there is no official Java library that corresponds to that annotations.proto file, you need to generate your own or do some shenanigans to modify the compiled descriptors before generating the Java code. I'll explain both.

    1. You can generate Java code for annotations.proto, not just your own proto that imports it.

      If you use Buf, you can actually tell it to generate source code for your imports using a --include-imports flag to buf generate (though this generates sources for all imports, not just ones that don't otherwise have a corresponding Java library).

      These annotations are available in Buf's Schema Registry, so you could also separately generate these files and compile them into a separate JAR using buf generate buf.build/grpc-ecosystem/grpc-gateway.

      One issue with this file is that it does not declare a Java package option in the file. That's why the Java package in the generated code doesn't have a proper reverse domain name. With Buf, you could use managed mode to actually inject a Java package option, to generate these files into whatever package you want (so you could generate them as if they were in a "shaded" package in your own JAR).

    2. This second route is much less advised and not for the faint of heart, but it lets you omit the import in the generated code. You need to first compile your sources to a file descriptor set. (Buf can produce this via buf build; With protoc, use a -o option instead of --java_out.) This file is a binary encoded FileDescriptorSet.

      You could write something that reads this file (unmarshalling its contents into a FileDescriptorSet) and then modifies it. You'd modify it by examining the dependency field of every file in the set and removing the entries like "protoc-gen-openapiv2/options/annotations.proto".

      You can then re-marshal this to a file and feed that back in for code generation. So instead of generating Java code from sources, you'd generate them from the modified descriptor set (buf generate <file>#format=bin if using Buf or protoc --descriptor_set_in=<file> --java_out=<output-dir> if using protoc).

      Note that this approach can only work if the only things that use the import being removed are custom options. That's because custom options can safely be represented as unrecognized fields in the descriptor (and effectively ignored). If you remove an import that has type definitions that are referenced in the file, the compiler will not accept the modified file descriptor set.

    If that last bullet looks like Greek, it's because that is quite advanced descriptor-fiddling. Realistically, I think the first bullet is your best approach.