DRAFT, NOT READY FOR PUBLICATION: Protocol Buffers should have asymmetric types

Max Bo
Adding more asymmetric types to Protocol Buffers - types that generate code that have stronger constructor validations than their deserialization validations - would make schema evolution safer.

Context

On the , the required and optional field labels were removed from Protocol Buffers, marking the release of Protocol Buffers v3.0.0. From now on, all fields would effectively behave as if they were annotated optional. While you would have to opt-in to the new syntax with an explicit syntax directive: 

syntax = "proto3";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
}

This change still upset quite a few people. On the , a Protocol Buffers maintainer offered up some rationale for it:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility. For example, if you add a required field to a .proto definition, binaries built with the new definition won't be able to parse data serialized using the old definition because the required field is not present in old data. In a complex system where .proto definitions are shared widely across many different components of the system, adding/removing required fields could easily bring down multiple parts of the system. We have seen production issues caused by this multiple times and it's pretty much banned everywhere inside Google for anyone to add/remove required fields. For this reason we completely removed required fields in proto3.

After the removal of required, optional is just redundant so we removed optional as well.

xfxyjwf; formatting added for readability

Because of this change, you see the following convention littered throughout proto3-using codebases: 

message ButtonPressRequest {
  // Required: which button to press
  Button button = 1;

  // Required: duration in milliseconds for how long to press the button
  uint32 duration_ms = 2;
}

In 2022, I joined Canva. For the next 3 years I worked on, for all intents and purposes, a hard fork of Protocol Buffers. This fork was locked to proto2 syntax, preserving the required and optional labels while adding a set of "Safe proto evolution extensions". This, along with a custom compatibility checker, almost entirely eliminated API and RPC compatibility issues at Canva.

In light of this, I view the proto3 changes to be one of software engineering's "throwing the baby out with the bathwater" moments, and generally a bad decision. I am writing this article to show that that some small extensions to proto2 or Protobuf Editions (and more broadly, any mature IDL, like Thrift or Smithy, etc.) can expand backwards compatibility guarantees. Ideally, Buf would introduce similar changes into Protovalidate and buf breaking.

Preamble

First, a refresher on Protocol Buffers , and some article specific terminology.

A proto file may define services, which define methods, that have request types and response types:

service SearchService {
  rpc Search(SearchRequest) returns (SearchResponse);
}

A proto file may define messages, that have fields with types and labels. Labels may indicate optional, required or repeated:†

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
}

A file may define enums, that have values:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

A committed change to a file is called an evolution:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
}

File versions can be generated into classes which build instances which serialize to payloads. Classes are built into artifacts and deployed into releases:‡

File versions, releases, instances, and payloads A file has three versions. Each file version generates classes, classes are built into artifacts, artifacts are deployed into releases, release classes construct or deserialize into instances, release instances serialize payloads, and payloads are sent to neighboring release classes. a file file version v0 file version v1 file version v2 classes classes classes artifacts artifacts artifacts a release classes instances a release classes instances a release classes instances payloads payloads payloads has has has generates generates generates built into built into built into construct or deserialize into construct or deserialize into construct or deserialize into deployed into deployed into deployed into serialize serialize serialize sent to sent to sent to sent to

We will use a simplified form of this diagram from now on.

A definition can be said to made "narrower" than its previous version when: 

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
  optional int32 results_per_page = 4;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

Conversely, a definition can can be made "wider" than its previous version when:

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional Corpus corpus = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

A method is made narrower if its request or response is made narrower.

A method is made wider if its request or response is made wider.

A service is made narrower if any its methods are made narrower.

A service is made wider if any its methods are made wider.

Problems with symmetric types

Assume an environment where we do not control deployment ordering, such for a given release, producers or consumers may be deployed before each other.

If an engineer needs to narrow a message, such as trying to add a required field they create an incompatibility risk. A message sent by an old producer breaks a new consumer:

message SearchRequest {
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
}
message SearchRequest {
  required string query = 1;
}

This also occurs when widening a message, by removing a required field. A message sent by a new producer breaks an old consumer:

message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}
message SearchRequest {
  required string query = 1;
}

This incompatibility risk doesn't just happen when adding or removing a required field directly: the same kind of risk occurs when transitioning an optional field to required, or required to optional.

Enum evolutions also carry incompatibility risk, when the enum is used as a required field, for example, when narrowing an enum, by removing an enum value. An enum sent by an old producer breaks a new consumer:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}

And also when widening an enum, by adding a value. An enum sent by a new producer breaks an old consumer:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

One may think that controlling the ordering of deploys prevents these incompatibilities, such that new consumers are released before new producers, or vice versa.

Not true.

A simultaneous narrowing and widening of a definition "deadlocks" a definition such there is no deployment ordering that prevents incompatibilities:

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORUPUS_AUDIO = 3;
}

Furthermore, if the evolution takes place in a domain definition that is referenced transitively by 2 separate messages that are in any method's request position and any method's response position, we also deadlock.

message User {
  optional string email = 1;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
  required Timestamp created_at = 2;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}
message User {
  optional string email = 1;
  required Timestamp created_at = 2;
}

message CreateUserRequest {
  required User user = 1;
}

message GetUserResponse {
  required User user = 1;
}

Asymmetric types

Asymmetric types admit safe evolution pathways that symmetric types do not, by having different validations on their generated class's constructor than its deserializer.

These validations execute at typecheck-time or runtime in the constructor, or just runtime in the deserializer.

A rule of thumb is that the constructor validation must always entail the deserialization validation.

More formally, we can say that:

\[ \mathrm{CV} \Rrightarrow \mathrm{DV} \]

which expands to:

\[ \forall x.\; \mathrm{CV}(x) \implies \mathrm{DV}(x) \]

Tristates

TODO: a differentiation between absence and non-recognition

Asymmetric types that could be supported

"constructor required deserialization optional"

Let's say the restrictions associated with the use of an explicit field default are too much: what are we to do?

In vanilla Protocol Buffers, unfortunately, nothing: there is no other safe evolution pathway to add or remove a required field without incompatibility risk as per the Problems with symmetric types section.

As stated before, I strongly believe the removal of optional was unnecessary, and an additional asymmetric label should have been added instead.

In Typical, the asymmetric label forces writers to provide a non-null value at the construction site but permits readers to deserialize payloads where that value is absent.

Using the email API example from Typical's tutorial, we can add a new from field to SendEmailRequest via a safe intermediate state:

struct SendEmailRequest {
    to: String = 0
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}
struct SendEmailRequest {
    to: String = 0
    from: String = 3
    subject: String = 1
    body: String = 2
}

This same intermediate state may also permit:

By looking at the generated code, we can see how this is achieved. Compiling the following Typical schema with typical generate: 

struct SendEmailRequest {
    to: String = 0
    asymmetric from: String = 3
    subject: String = 1
    body: String = 2
}

In Typical, fields are required by default, so to, subject, and body do not need an explicit label.

produces two distinct Rust types: a SendEmailRequestOut used to construct and serialize a message, and a SendEmailRequestIn produced by deserialization. The unlabeled to, subject, and body fields remain required in both directions. The asymmetric label makes from a non-optional String on the way out, but an Option<String> on the way in: 

pub struct SendEmailRequestOut {
    pub to: String,
    pub from: String,
    pub subject: String,
    pub body: String,
}

pub struct SendEmailRequestIn {
    pub to: String,
    pub from: Option<String>,
    pub subject: String,
    pub body: String,
}

This is the asymmetry made concrete: the constructor demands a value, while the deserializer tolerates its absence. The generated From<SendEmailRequestOut> for SendEmailRequestIn conversion simply wraps the value in Some: 

impl From<SendEmailRequestOut> for SendEmailRequestIn {
    fn from(message: SendEmailRequestOut) -> Self {
        SendEmailRequestIn {
            to: message.to.into(),
            from: Some(message.from.into()),
            subject: message.subject.into(),
            body: message.body.into(),
        }
    }
}

In Protocol Buffers, we could express the same idea with a new field option, constructor_required. For this option to be useful, generated Protocol Buffers construction APIs would need to change significantly: instead of defaulting to a no-argument builder and checking required fields later, producer-facing construction would need to require constructor fields up front. The from field is still optional for deserialization, but generated producer-facing construction APIs require it: 

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  optional bool constructor_required = 50002;
}

message SendEmailRequest {
  required string to = 1;
  required string subject = 2;
  required string body = 3;
  optional string from = 4 [(constructor_required) = true];
}

Compared to default Java generation, the producer-facing builder entry point changes like this:

public final class SendEmailRequest {
  // ...

  public boolean hasFrom() {
    // ...
  }

  public String getFrom() {
    // ...
  }

  public static Builder newBuilder() {
    return new Builder();
  }

  public static Builder newBuilder(
      String to,
      String subject,
      String body,
      String from) {
    return new Builder(to, subject, body, from);
  }

  public static final class Builder {
    private Builder() {
      // ...
    }

    private Builder(String to, String subject, String body, String from) {
      // to, subject, body, and from are required for producer-side construction.
      // ...
    }

    public SendEmailRequest build() {
      // ...
    }
  }

  static SendEmailRequest parseFromWire(byte[] payload) {
    // May return a message where hasFrom() is false.
    // ...
  }
}
SendEmailRequest.newBuilder();
// Does not compile: expected required constructor fields.

SendEmailRequest.newBuilder(
        "typical@example.com",
        "I love Typical!",
        "It makes serialization easy and safe.",
        "newsletter@example.com")
    .build(); // ok

Unconstructable variants

An enum value option that prevents the construction of a specific enum value, while still permitting its deserialization, turns the enum into an asymmetric type.

The option permits us to safely add a variant under the assumption that no other engineer can construct, serialize and send it to deserializers that are unprepared to handle it.

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}
enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3;
}

We could express this in Protocol Buffers as a custom enum value option:

import "google/protobuf/descriptor.proto";

extend google.protobuf.EnumValueOptions {
  optional bool unconstructable = 50001;
}

enum Corpus {
  CORPUS_UNSPECIFIED = 0;
  CORPUS_TEXT = 1;
  CORPUS_IMAGES = 2;
  CORPUS_AUDIO = 3 [(unconstructable) = true];
}

This would generate Java code that looks like:

public enum Corpus {
  CORPUS_UNSPECIFIED,
  CORPUS_TEXT,
  CORPUS_IMAGES,
  CORPUS_AUDIO; // [(unconstructable) = true]

  public static Corpus fromWireValue(int number) {
    // ...
  }
}

public enum ConstructableCorpus {
  CORPUS_UNSPECIFIED,
  CORPUS_TEXT,
  CORPUS_IMAGES;

  Corpus asCorpus() {
    // ...
  }
}

public final class SearchRequest {
  // ...

  public Corpus getCorpus() {
    // ...
  }

  public static final class Builder {
    // ...

    public Builder setCorpus(ConstructableCorpus corpus) {
      // ...
    }
  }
}

With this shape, producer code cannot pass Corpus.CORPUS_AUDIO to the builder at all:

SearchRequest ok =
    SearchRequest.newBuilder()
        .setCorpus(ConstructableCorpus.CORPUS_TEXT)
        .build();

SearchRequest nope =
    SearchRequest.newBuilder()
        .setCorpus(Corpus.CORPUS_AUDIO)
        // Does not compile:
        // required ConstructableCorpus, found Corpus
        .build();

But CORPUS_AUDIO is still a real case/value that generated and handwritten reader code can handle exhaustively:

switch (request.getCorpus()) {
  case CORPUS_UNSPECIFIED -> handleUnspecified();
  case CORPUS_TEXT -> handleText();
  case CORPUS_IMAGES -> handleImages();
  case CORPUS_AUDIO -> handleAudio();
}

Two validations

Some IDLs support arbitrary field validations, such as Amazon's Smithy and Protocol Buffers with Buf's protovalidate.

But these are symmetric validations and thus cannot change without incompatibility risk:

By narrowing a validation, we induce an incompatibility risk:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 10"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 10"
  ];
}

And also by widening the validation:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (cel_validation) = "this <= 30"
  ];
}

Once again, by decoupling the constructor validation and the deserialisation validation we admit the ability to change the validation:

Now we have safe narrowing:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 10"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 10",
    (deserialize_cel) = "this <= 10"
  ];
}

And widening:

message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 30",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 20"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 20",
    (deserialize_cel) = "this <= 30"
  ];
}
message SearchRequest {
  required string query = 1;
  required int32 page_number = 2 [
    (constructor_cel) = "this <= 30",
    (deserialize_cel) = "this <= 30"
  ];
}

For this to work, we must ensure that a message field's constructor validation must entail the deserialisation validation at all times. But checking whether this invariant holds, and whether a proposed evolution is legal, can prove difficult.

As an example, CUE can prove that evolutions of field validations are compatible with their previous file versions using its lattice-based type system.

Although somewhat overkill, I believe that Z3 would be sufficient to prove entailment, and to check that the constructor validation entails the deserialization validations of all previous file versions.

As a small demo, we ask Z3 for a counterexample to ∀x. CV(x) ⟹ DV(x) by asking it to find some x such that CV(x) ∧ ¬DV(x) is true.

# /// script
# requires-python = ">=3.10"
# dependencies = ["z3-solver"]
# ///
from z3 import Int, Not, Solver, sat


def check_entailment(cv, dv):
    x = Int("x")

    # Prove ∀x. CV(x) ⟹ DV(x) by asking for a counterexample:
    #   ∃x. CV(x) ∧ ¬DV(x)
    solver = Solver()
    solver.add(cv(x))
    solver.add(Not(dv(x)))

    if solver.check() == sat:
        witness = solver.model()[x]
        print(f"counterexample x = {witness}")
    else:
        print("safe")


# safe
check_entailment(lambda x: x <= 10, lambda x: x <= 20)

# counterexample x = 21
check_entailment(lambda x: x <= 30, lambda x: x <= 20)

Other asymmetric types

Choice fallbacks

The Typical IDL supports a structure called choice that supports a novel evolution mechanism when used with the asymmetric label: it strengthens the constructor validation such that it demands 2 values, but the deserialization validation only requires 1, the fallback.

The tutorial's email API example uses both kinds of fallback field:

choice SendEmailResponse {
    success = 0
    error: String = 1

    # A more specific type of error for curious clients
    optional authentication_error: String = 2

    # To be promoted to required in the future
    asymmetric please_try_again = 3
}

Adding and removing the optional authentication_error case is safe because writers provide a fallback, and readers that do not know or do not care about that case can use the fallback instead.

choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    optional authentication_error: String = 2
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    optional authentication_error: String = 2
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }

Adding the asymmetric please_try_again case is the dual of the struct example above: writers may choose the new case only if they also include a fallback, while readers that know the asymmetric case must handle it directly. Once all readers can handle it, the field can be promoted to required.

choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    asymmetric please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    asymmetric please_try_again = 3
  }
choice SendEmailResponse {
    success = 0
    error: String = 1
    please_try_again = 3
  }

The generated Rust shows how the fallback is enforced. The writer-side enum requires fallback values for the optional and asymmetric cases, while the reader-side enum only exposes the asymmetric case itself:

pub enum SendEmailResponseOut {
    Success,
    Error(String),
    AuthenticationError(String, Box<SendEmailResponseOut>),
    PleaseTryAgain(Box<SendEmailResponseOut>),
}

pub enum SendEmailResponseIn {
    Success,
    Error(String),
    AuthenticationError(String, Box<SendEmailResponseIn>),
    PleaseTryAgain,
}

A writer choosing PleaseTryAgain must supply a Box<SendEmailResponseOut>, a fallback case that older readers (which don't yet know please_try_again) can fall back to. A reader, on the other hand, sees a payload-free PleaseTryAgain. The generated conversion discards the fallback once the case is understood:

impl From<SendEmailResponseOut> for SendEmailResponseIn {
    fn from(message: SendEmailResponseOut) -> Self {
        match message {
            SendEmailResponseOut::Success => SendEmailResponseIn::Success,
            SendEmailResponseOut::Error(payload) => SendEmailResponseIn::Error(payload.into()),
            SendEmailResponseOut::AuthenticationError(message, fallback) => {
                SendEmailResponseIn::AuthenticationError(
                    message.into(),
                    Box::new((*fallback).into()),
                )
            }
            SendEmailResponseOut::PleaseTryAgain(_fallback) => SendEmailResponseIn::PleaseTryAgain,
        }
    }
}