Using the python protobuf library, how can I load a .desc (or proto) file dynamically and reflect the fields within

Question

I've tried using a few different methods but all end up not working.

I'm trying to read from a directory and iterate all the .desc files and inspect all the fields in each class. I'm trying to build a dependency tree with the outer classes being the parent nodes and the leaves being the nested classes. I'm trying to do this all dynamically based on a root folder containing the .desc files (or proto files). Nothing is statically typed.

The error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 197: 'utf-8' codec can't decode byte 0xf8 in position 197: invalid start byte in field: google.protobuf.FileDescriptorProto.name

def gen1(desc_file, db=None):
if not db:
    db = symbol_database.Default()

with open(desc_file, 'rb') as fh:
    fds = _descriptor_pool.DescriptorPool().AddSerializedFile(fh.read()) # <- this errors
generated = {}

for prot in fds.file:
    fd = db.pool.Add(prot)
    for name in fd.message_types_by_name:
        mdesc = fd.message_types_by_name[name]
        Klass = reflection.MakeClass(mdesc)
        generated[(fd.package, name)] = db.RegisterMessage(Klass)

return generated

Answer 1

The Protobuf hierarchy is confusing/complex.

The following is a basic approach to enumerating the contents of a Protobuf descriptor file:

from google.protobuf.descriptor_pb2  import FileDescriptorSet


with open("descriptor.pb", 'rb') as descriptor:
    data = descriptor.read()
    fds = FileDescriptorSet.FromString(data)

    # file: FileDescriptorProto
    for file in fds.file:
        print(f"{file.package}/{file.name}")

        # service: ServiceDescriptorProto
        for service in file.service:
            print(service.name)

            # method: MethodDescriptorProto
            for method in service.method:
                print(method.name)
             
        # message: DescriptorProto
        for message_type in file.message_type:
            print(message_type.name)
            # field: FieldDescriptorProto
            for field in message_type.field:
                print(field.name)