DRAFT, NOT READY FOR PUBLICATION: Protocol Buffers should have asymmetric types

Max Bo
Adding more asymmetric types to Protocol Buffers - types that generate code that have stronger constructor validations than their deserialization validations - would make schema evolution safer.

Context

On the , the required and optional field labels were removed from Protocol Buffers, marking the release of Protocol Buffers v3.0.0. From now on, all fields would effectively behave as if they were annotated optional. While you would have to opt-in to the new syntax with an explicit syntax directive: 

syntax = "proto3";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
}

this change still upset quite a few people. On the , a Protocol Buffers maintainer offered up some rationale for it, which I encourage you to read in full:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility. For example, if you add a required field to a .proto definition, binaries built with the new definition won't be able to parse data serialized using the old definition because the required field is not present in old data. In a complex system where .proto definitions are shared widely across many different components of the system, adding/removing required fields could easily bring down multiple parts of the system. We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3.

After the removal of required, optional is just redundant so we removed optional as well.

xfxyjwf; formatting added for readability

In 2022, I joined Canva. For the next 3 years I worked on, for all intents and purposes, a hard fork of Protocol Buffers. This fork was locked to proto2 syntax, preserving the required and optional labels while adding a set of "Safe proto evolution extensions". This, along with a custom compatibility checker, almost entirely eliminated API and RPC compatibility issues at Canva.

In light of this, I view the proto3 changes to be one of software engineering's "throwing the baby out with the bathwater" moments, and generally a bad decision. I am writing this article to show that that some small extensions to proto2 or Protobuf Editions (and more broadly, any mature IDL, like Thrift or Smithy, etc.) can expand backwards compatibility guarantees. Ideally, Buf would introduce similar changes into Protovalidate and the buf breaking compatibiity checker..

Why we should want required

proto3 shifts the responsibility to reject payloads with missing data out of the IDL instance deserialization logic into domain logic. Because of this, you will see the following convention littered throughout proto3-using codebases as a reminder to perform this validation: 

message ButtonPressRequest {
  // Required: which button to press
  Button button = 1;

  // Required: duration in milliseconds for how long to press the button
  uint32 duration_ms = 2;
}

Thus, the design of a proto3 stands as a rejection of a "parse, don't validate" philosophy.

Preamble

First, a refresher on Protocol Buffers, specifically proto2, and some article specific terminology.

A .proto file may define services, which define methods, that have request types and response types:

service SearchService {
  rpc Search(SearchRequest) returns (SearchResponse);
}

A .proto file may define messages, that have fields with types and labels. Labels may indicate optional, required or repeated:†

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
}

A .proto file may define enums, that have values:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

A committed change to a file is called an evolution:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
}

File versions can be generated into classes which build instances which serialize to payloads. Classes are built into artifacts and deployed into releases:‡

File versions, releases, instances, and payloads A file has three versions. Each file version generates classes, classes are built into artifacts, artifacts are deployed into releases, release classes construct or deserialize into instances, release instances serialize payloads, and payloads are sent to neighboring release classes. a file file version v0 file version v1 file version v2 classes classes classes artifacts artifacts artifacts a release classes instances a release classes instances a release classes instances payloads payloads payloads has has has generates generates generates built into built into built into construct or deserialize into construct or deserialize into construct or deserialize into deployed into deployed into deployed into serialize serialize serialize sent to sent to sent to sent to

We will use a simplified form of this diagram from now on.

A definition can be said to made "narrower" than its previous version when: 

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
  optional int32 results_per_page = 4;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

Conversely, a definition can can be made "wider" than its previous version when:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

Problems with symmetric types

Let's dig further into the problems associated with required described by the quote in the context section.

Assume an environment where we do not control deployment ordering, such for a given release, producers or consumers may be deployed before each other.

If an engineer needs to narrow a message, such as trying to add a required, field they create an incompatibility risk. A message sent by an old producer can break a new consumer:

message SearchRequest {
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
}
message SearchRequest {
  required string query = 1;
}

This also occurs when widening a message, by removing a required field. A message sent by a new producer can break an old consumer:

message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}

This incompatibility risk doesn't just happen when adding or removing a required field directly: the same kind of risk occurs when evolving an optional field to required, or required to optional.

Enum evolutions also carry incompatibility risk, when the enum is used as a required field, for example, when narrowing an enum, by removing an enum value. An enum sent by an old producer can break a new consumer:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

And also when widening an enum, by adding a value. An enum sent by a new producer break can break an old consumer:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

One may think that controlling the ordering of deploys prevents these incompatibilities, such that new consumers are released before new producers, or vice versa.

Not true.

A simultaneous narrowing and widening of a definition "deadlocks" a definition such there is no deployment ordering that prevents incompatibilities:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

Furthermore, if the evolution takes place in a domain definition that is referenced transitively by 2 separate messages that are in any method's request position and any method's response position, we also deadlock.

message User {
  optional string email = 1;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
  required Timestamp created_at = 2;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
  required Timestamp created_at = 2;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}

Asymmetric types

Asymmetric types admit safe evolution pathways that symmetric types do not, by having different validations on their generated class's constructor than its deserializer.

These validations execute at typecheck-time or runtime in the constructor, or just runtime in the deserializer.

A rule of thumb is that the constructor validation must always entail the deserialization validation.

More formally, we can say that:

\[ \mathrm{CV} \Rrightarrow \mathrm{DV} \]

which expands to:

\[ \forall x.\; \mathrm{CV}(x) \implies \mathrm{DV}(x) \]

Tristates

TODO: a differentiation between absence and non-recognition

Asymmetric types that could be supported

"constructor required deserialization optional"

Thus, the second, less common type of asymmetric type we'll look at is a "constructor required deserialization optional" type.

† label.

In Typical adding the asymmetric label to a field generates code treats the field as required in the constructor, such that it must be parameterized, whilst treating it as optional in its deserializer and by public field access.

Using the example from Typical README, we can see that by first adding field from with anasymmetric label, we can safely evolve it it to a (labelless) required.

struct SendEmailRequest {
    to: String = 0
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    from: String = 3
    subject: String = 1
    body: String = 2
}

How wonderful! Using asymmetric as this intermediate stage admits:

Before moving on, I think it's worth looking at what asymmetric does to the generated Rust code: 

struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}
pub struct SendEmailRequestOut {
    pub to: String,
    pub from: String,
    pub subject: String,
    pub body: String,
}

pub struct SendEmailRequestIn {
    pub to: String,
    pub from: Option<String>,
    pub subject: String,
    pub body: String,
}
impl From<SendEmailRequestOut> for SendEmailRequestIn {
    fn from(message: SendEmailRequestOut) -> Self {
        SendEmailRequestIn {
            to: message.to.into(),
            from: Some(message.from.into()),
            subject: message.subject.into(),
            body: message.body.into(),
        }
    }
}

Porting this behaviour to proto2 is not trivial.

It is simple 

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  optional bool constructor_required = 50002;
}

message SendEmailRequest {
  required string to = 1;
  required string subject = 2;
  required string body = 3;
  optional string from = 4 [(constructor_required) = true];
}

Compared to default Java generation, the producer-facing builder entry point changes like this:

public final class SendEmailRequest {
  // ...

  public boolean hasFrom() {
    // ...
  }

  public String getFrom() {
    // ...
  }

  public static Builder newBuilder() {
    return new Builder();
  }

  public static Builder newBuilder(
      String to,
      String subject,
      String body,
      String from) {
    return new Builder(to, subject, body, from);
  }

  public static final class Builder {
    private Builder() {
      // ...
    }

    private Builder(String to, String subject, String body, String from) {
      // to, subject, body, and from are required for producer-side construction.
      // ...
    }

    public SendEmailRequest build() {
      // ...
    }
  }

  static SendEmailRequest parseFromWire(byte[] payload) {
    // May return a message where hasFrom() is false.
    // ...
  }
}
SendEmailRequest.newBuilder();
// Does not compile: expected required constructor fields.

SendEmailRequest.newBuilder(
        "TODO",
        "TODO",
        "TODO",
        "TODO")
    .build(); // ok

Unconstructable variants

An enum value option that prevents the construction of a specific enum value, while still permitting its deserialization, turns the enum into an asymmetric type.

This option permits us to safely add a variant under the assumption that no other engineer can construct, serialize and send it to deserializers that are unprepared to handle it.

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3;
}

We could express this in Protocol Buffers as a custom enum value option:

import "google/protobuf/descriptor.proto";

extend google.protobuf.EnumValueOptions {
  optional bool unconstructable = 50001;
}

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}

This would generate Java code that looks like:

public enum Corpus {
  CORPUS_UNSPECIFIED,
  CORPUS_TEXT,
  CORPUS_IMAGES,
  CORPUS_AUDIO; // [(unconstructable) = true]

  public static Corpus fromWireValue(int number) {
    // ...
  }
}

public enum ConstructableCorpus {
  CORPUS_UNSPECIFIED,
  CORPUS_TEXT,
  CORPUS_IMAGES;

  Corpus asCorpus() {
    // ...
  }
}

public final class SearchRequest {
  // ...

  public Corpus getCorpus() {
    // ...
  }

  public static final class Builder {
    // ...

    public Builder setCorpus(ConstructableCorpus corpus) {
      // ...
    }
  }
}

This would prevent passing Corpus.CORPUS_AUDIO to the builder:

SearchRequest ok =
    SearchRequest.newBuilder()
        .setCorpus(ConstructableCorpus.CORPUS_TEXT)
        .build();

SearchRequest nope =
    SearchRequest.newBuilder()
        .setCorpus(Corpus.CORPUS_AUDIO)
        // Does not compile:
        // required ConstructableCorpus, found Corpus
        .build();
switch (request.getCorpus()) {
  case CORPUS_UNSPECIFIED -> ...;
  case CORPUS_TEXT -> ...;
  case CORPUS_IMAGES -> ...;
  case CORPUS_AUDIO -> ...;
}

Two validations

Some IDLs support arbitrary field validations, such as Amazon's Smithy and Protocol Buffers with Buf's protovalidate.

But these are symmetric validations and thus cannot change without incompatibility risk:

By narrowing a validation, we induce an incompatibility risk:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 10"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 10"
  ];
}

And also by widening the validation:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 30"
  ];
}

Once again, by decoupling the constructor validation and the deserialisation validation we admit the ability to change the validation:

Now we have safe narrowing:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 10"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 10"
  ];
}

And widening:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 30",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 30",
    (deserialize_cel) = "this <= 30"
  ];
}

For this to work, we must ensure that a message field's constructor validation must entail the deserialisation validation at all times. But checking whether this invariant holds, and whether a proposed evolution is legal, can prove difficult.

As an example, CUE can prove that evolutions of field validations are compatible with their previous file versions using its lattice-based type system.

Although somewhat overkill, I believe that Z3 would be sufficient to prove entailment, and to check that the constructor validation entails the deserialization validations of all previous file versions.

As a small demo, we ask Z3 for a counterexample to ∀x. CV(x) ⟹ DV(x) by asking it to find some x such that CV(x) ∧ ¬DV(x) is true.

# /// script
# requires-python = ">=3.10"
# dependencies = ["z3-solver"]
# ///
from z3 import Int, Not, Solver, sat


def check_entailment(cv, dv):
    x = Int("x")

    # Prove ∀x. CV(x) ⟹ DV(x) by asking for a counterexample:
    #   ∃x. CV(x) ∧ ¬DV(x)
    solver = Solver()
    solver.add(cv(x))
    solver.add(Not(dv(x)))

    if solver.check() == sat:
        witness = solver.model()[x]
        print(f"counterexample x = {witness}")
    else:
        print("safe")


# safe
check_entailment(lambda x: x <= 10, lambda x: x <= 20)

# counterexample x = 21
check_entailment(lambda x: x <= 30, lambda x: x <= 20)

Other asymmetric types

Choice fallbacks

The Typical IDL supports a structure called choice that supports a novel evolution mechanism when used with the asymmetric label: it strengthens the constructor validation such that it demands 2 values, but the deserialization validation only requires 1, the fallback.

The tutorial's email API example uses both kinds of fallback field:

choice SendEmailResponse {
    success = 0
    error: String = 1

    # A more specific type of error for curious clients
    optional authentication_error: String = 2

    # To be promoted to required in the future
    asymmetric please_try_again = 3
}
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    optional authentication_error: String = 2
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    optional authentication_error: String = 2
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    asymmetric please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    asymmetric please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    please_try_again = 3
  }
pub enum SendEmailResponseOut {
    Success,
    Error(String),
    AuthenticationError(String, Box<SendEmailResponseOut>),
    PleaseTryAgain(Box<SendEmailResponseOut>),
}

pub enum SendEmailResponseIn {
    Success,
    Error(String),
    AuthenticationError(String, Box<SendEmailResponseIn>),
    PleaseTryAgain,
}
impl From<SendEmailResponseOut> for SendEmailResponseIn {
    fn from(message: SendEmailResponseOut) -> Self {
        match message {
            SendEmailResponseOut::Success => SendEmailResponseIn::Success,
            SendEmailResponseOut::Error(payload) => SendEmailResponseIn::Error(payload.into()),
            SendEmailResponseOut::AuthenticationError(message, fallback) => {
                SendEmailResponseIn::AuthenticationError(
                    message.into(),
                    Box::new((*fallback).into()),
                )
            }
            SendEmailResponseOut::PleaseTryAgain(_fallback) => SendEmailResponseIn::PleaseTryAgain,
        }
    }
}