Search code examples
protocol-buffersprotoc

How to get protoc to compile a proto with its dependencies?

I need to fix a grpc service, so I want to understand the logic around compiling them.

In the example below, I don't understand why protoc doesn't compile the address.proto, since it's imported by person.proto.

There are no build errors, so I don't think it's an import/naming issue. Unfortunately, only a person_pb2.py is generated...

// file: address.proto

syntax="proto3";

message Address {
    string city = 1;
    string road = 3;
    int32 roadNumber = 4;
}

// file: person.proto

syntax="proto3";

import "address.proto";

message Person {
  string name = 1;
  Address home = 3;
  Address work = 4;
}

build command:

python -m grpc_tools.protoc --proto_path ../protos --python_out=. person.proto
Solution

  • It's a reasonable question.

    The answer is that you're only asking protoc to compile person.proto (and it's taking you literally). But, the generated code won't run because person_pb2.py depends on address_pb2:

    import address_pb2 as address__pb2

    You need to have Python sources for all protocol buffers types that your code needs.

    python3 \
-m grpc_tools.protoc \
--proto_path=../protos \
--python_out=. \
person.proto address.proto

    A good example of this problem (being solved) is Google's so-called Well-Known Types (WKTs)

    You could include e.g. google.protobuf.Timestamp in Person by adding import "google/protobuf/timestamp.proto":

    syntax="proto3";

import "google/protobuf/timestamp.proto";
import "address.proto";

message Person {
  string name = 1;
  Address home = 3;
  Address work = 4;
  google.protobuf.Timestamp timestamp = 5;
}

    And you can protoc without change and your code will work.

    This is because Google bundles the WKTs and generated Python code with grpcio-tool. As with any other source, you do need to import the Google-provided Python sources to use the code.

    To do this, you must

    main.py:

    import address_pb2
import person_pb2

from google.protobuf.timestamp_pb2 import Timestamp

    So, where are these files?

    In my case:

    lib/python3.8/site-packages/grpc_tools/_proto/google/protobuf/timestamp.proto

    And:

    lib/python3.8/site-packages/google/protobuf/timestamp_pb2.py

    NOTE
    It's convenient that Google bundles the generated sources but this is not required and it creates two potential problems.

    1. timestamp.proto is the source-of-truth and we must assume|trust that google/protobuf/timestamp_pb2.py was generated from it.
    2. If we're the proto developer, we're expected to produce sources for all languages that our proto's developers may use every time we update the protos.

    For these reasons, generally (!) developers provide only the protos and don't include generated sources and assume that you're able to plug the proto into protoc and generated the perfect facsimile as code yourself.