Google Cloud Pub/Sub Schemas - Too many protobuf messages in single schema

Question

I'm defining a schema for a pub/sub topic.

I have 7 proto files, and I'm able to combine them. When I did this, GCP outputted that there were too many protobuf messages in a single schema. I believe this is something to do with the 50Kb schema definition limit. I have looked at the related stack thread, but I am not able to combine my messages within other messages since they need to be used in multiple other places.

If I was to make 1 schema for each of my protobuf files, would I be able to use multiple schemas for a single topic? Or would I have to combine them (which has thrown me the above error).

Answer 1

Pub/Sub proto schemas can only contain a single top-level message definition. In other words, a schema such as the following would return the error you see:

message M1 {
  string f1 = 1;
}

messages M2 {
  M1 m1 = 1;
}

In such a case, you'd need to nest the definitions:

message M2 {
  message M1 {
    string f1 = 1;
  }
  M1 m1 = 1;
}

This is noted in the Pub/Sub schema documentation. It also mentions that imports are not allowed, which is why you are getting the error about trying to import the timestamp proto.

A topic can only only have a single schema associated with it, so you cannot separate these into multiple schemas and use them all in the same topic. You could consider defining a single schema that has a union of the types you wish to use:

message FullSchema {
  message M1 {
    string f1 = 1;
  }

  message M2 {
    string f1 = 1;
  }

  oneof value {
    M1 m1 = 1;
    M2 m2 = 2;
  }
}