Search code examples
goprotocol-buffersbackwards-compatibilityproto3

Can I add "optional" to an existing primitive field in proto3? Is it backwards compatible for Golang code?

I have a field in a proto3 message that is currently a primitive type. To differentiate between 0 (the default value) and an unset state, I am considering marking the field as optional.

Here’s my question:

  1. Is adding optional to an existing primitive field in proto3 considered backwards compatible?
    For example:

    syntax = "proto3";

message Example {
    int32 existing_field = 1; // Current field
}

    Updating to:

    syntax = "proto3";

message Example {
    optional int32 existing_field = 1; // Marking it optional
}

  2. I am generating Golang code from this proto file. Since marking a field as optional in proto3 causes the generated Go field to become a pointer, will this break existing code using the field?

  3. If adding optional is not backwards compatible, is deprecating the existing field and creating a new one the correct approach?

Any guidance on how to handle this scenario while maintaining backward compatibility would be greatly appreciated!

Solution

  • If this changes the generated code: then obviously you'll need to fixup any consuming code.

    At the data protocol level, there are some implications here:

    • without optional, this uses zero===default logic, so: if a value is zero whether explicitly or implicitly, no value is transmitted; non-zero values are transmitted
    • with optional, it uses assignment tracking, so: if a value is explicitly assigned, it is transmitted (even if zero); otherwise (if not assigned): the value is not transmitted

    This can have some impact if you are either storing payloads or transmitting payloads between system that are not updated at exactly the same time. There are a number of ways where the above could lead to confusion as to whether a value has an explicit or not - the most obvious being an explicitly assigned zero in the old code being interpreted as no value assigned in new code. If your code is going to handle that possibility: then fine.

    Yes, marking the old field as deprecated and adding a new field is also an option, as long as you handle old payloads with the old field, if storing payloads or transmitting payloads between the two systems. However, make sure to consider whether it is possible for data to be serialized using the new layout and deserialized/received by a system that only knows about the old layout - as in that case, the new field will be ignored, and the old field will presumably be zero. This may lead to confusion.