DRAFT, NOT READY FOR PUBLICATION: Interface Description Languages should support more asymmetric types

Max Bo

IDLs that support asymmetric types generate code that have stronger constructor validations than their deserialization validations. I believe they make schema evolution safer and more IDLs should support them.

Context

I worked at Canva for 3 and a bit years, where I worked on, for all intents and purposes, a hard fork of Protocol Buffers. One addition in that fork was that of the "Safe proto evolution extensions", that, along with a schema registry, almost entirely eliminated API and RPC compatibility issues.

Terminology

For the purposes of this article, I will define some terminology upfront. If may be non-standard as I'm not entirely around the academic literature, for which I apologize.

use mermaid

flowchart TD
    A[Schema (abstract)]
    B[Schema version (abbreviated just version) (concrete file)]
    C[(generate) classes]
    D[(validate and construct) instances]
    E[(serialize) payloads]
    F[(built into) artifacts]
    G[(deployed as) Deployment]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G

Schemas may be services, which define methods, that have request types and response types

They may be messages, that have fields with types and labels. Labels may indicate optional, 'required' or 'repeated'

I make a distinction between a "type", in an IDL, and a "validation" in generated code. Typically a type generates 1 validation. In the case of asymmetric validations, 2.

They may be enums, that have values.

Schemas generate classes which build instances which serialize to payloads. Objects are embedded in artifacts and deployed into old releases.

Out of convenience, I may choose mention a payload version rather than a "payload serialized by an instance build by a class generated by a specific schema version".

Versions can have a "liveness" property. A version is live if its classes CAN attempt to deserialize a new payload OR a version's payload CAN be deserialized by new classes.

This implies that if a specific schema version's payload is persisted to a data store, it remains indefinitely live.

A change to a schema is called an evolution.

A schema can be said to made "narrower" than its previous version when:

For messages (a product type), we add a unique new field
For enums (a sum type), we add a unique new value

Conversely, a schema can can be made "wider" than its previous version when:

for messages, we remove a field
for enums, we remove a value

A method, is said to be made narrower if its request or response is made narrower. (Repeat)

A service is said to made narrower if any its methods are made narrower (etc)

For the purposes of this article I will use proto2 syntax and semantics and Java, but I believe this would applicable to any mature IDL (list them) and language target. I will highlight suggested modifications in bold. I will highlight non-existent, but demonstration extensions, in italics.

Problems with asymmetric types

We assume an environment where we do not control deployment ordering, such for a given release.

If an engineer needs to strengthen a message, such as the simple act of a trying to add a required field they create an incompatibility risk.

This also occurs when weakening a message, by removing a field.

And strengthening an enum, by removing a value.

And weakening an enum, by adding a value.

One may assume that controlling the ordering of deploys prevents these incompatibilities.

Not true.

A simultaneous narrowing and widening of a schema "deadlocks" the schema such there is no deployment ordering that prevents incompatibilities.

Furthermore, if the evolution takes place in a domain object that is referenced transitively by 2 separate messages that are in a request position and a response position also deadlocks.

Even ordered deployment is insufficient to avoid this scenario.

What is an asymmetric type

Asymmetric types have different validations on their classes constructor than its deserializer.

These validations execute at typecheck-time or runtime in the constructor, or just runtime in the deserializer.

A rule of thumb is that the constructor validation must always entail the deserialization validation.

A compatibility checker can check, statically, whether a schema evolution will cause incompatibilities if the evolution is merged and then deployed immediately or at the next scheduled release.

More formally, CV ⇛ DV which expands to ∀x. CV(x) ⟹ DV(x)

There is a rich design space of asymmetric types. Let's look at a few.

Defaults

Field defaults

The most common form of asymmetric type is a field default.

It relaxes the just deserializer such that it accepts missing payload data.

But the default must be a member of type field type T.

This poses 2 problems:

Some types don't have a reasonable default value, particularly non-primitive messages.

We may typically choose a 0 default for primitives but we deprive the domain layer of useful distinction: whether the 0 was a fallback or a legitimate value sent over the wire.

Field fallbacks

typical has a novel evolution mechanism: it strengthens the constructor validation such it demands 2 values, but the deserialization validation only requires 1, the fallback.

Type defaults

Implicit type defaults

Protocol Buffers is particularly egregious when it comes to utilizing this defaulting behavior. Along with syntax for explicitly registering a default, it implicitly enrolls most primitive types into this defaulting behavior, and even enums, defaulting to the member with the lowest value. Thus, proto enums, are defined by programming convention to have an UNKNOWN variant, that either pollutes the domain model, or necessitate a validation post-deserialization to strip out the UNKNOWN variant and map it onto a UNKNOWN-less domain model enum.

I feel this rather common pattern is an unfortunate rejection of a "parse, don't validate" philosophy.

Explicit type defaults

Tristates

a differentiation between absence and non-recognition

Optional vs required

The simplest means of avoiding the use of a default is to use an optional label when adding fields.

But unfortunately, optional alone offers no such safe evolution pathway to transform a field into a required field. optional can simply not be transitioned into required without incompatibility risk, and vice versa.

DIAGRAM

Use of required was perceived to so fraught with danger at Google that the following tortured convention was littered throughout the google3 monorepo

message Foo {
  // required
  optional
}

and the required label was removed entirely in proto3 syntax.

I strongly believe the removal or this label was unnecessary, and instead an asymmetric label should have been added instead.

Thus, the second, less common type of asymmetric type we'll look at is "construct required deserialization optional" label. To save ink, I will instead call it asymmetric the same as field label from the IDL typical, and the namesake of this article.

asymmetric label forces parameterization of a non-null value at the constructor site but permits data absence in the payload.

Use of this label admits safe evolution semantics! We use it as an intermediate stage when:

transitioning from optional to required — DIAGRAM
adding a required — DIAGRAM
Transitioning from required to optional — DIAGRAM
Removing a required field — DIAGRAM

Unproducible

The third type is unproducible.

For enums (or any sum type), we prevent the use of the value or variant in certain contexts, but permit its deserialization.

The label permits us to safely add a variant under the assumption that no other engineer can construct, serialize and send it to deserializers that are unprepared to handle it.

One of the reasons I suspect that it quite difficult, although possible, to statically enforce unproducibility.

One such way is to enforce unproducibility is the use of a linter that ensures that values and constructors may only be used in 'if' tests, and switch case labels and patterns.

Two validations

The fourth type is "two validations".

Some IDLs support arbitrary field validations, such as Amazon's XXXX and Protocol Buffers with Buf's protovalidate.

But these validations inherit the restrictions placed upon required and optional fields: they cannot change safely in RPC and API contexts.

TO CLAUDE: https://protovalidate.com/schemas/custom-rules/#basics-of-cel-expressions Change these to use protovalidate syntax

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.cel_validation) =  "this <= 5"]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 4"]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 5",
       (validation_extensions.deserialization_cel_validation) = "this <= 5"]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 6",
       (validation_extensions.deserialization_cel_validation) = "this <= 6"]}

Once again, decoupling the constructor validation and the deserialisation validation admits the ability to change the validation.

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 5",
       (validation_extensions.deserialization_cel_validation) = "this <= 5"]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 5",
       (validation_extensions.deserialization_cel_validation) = "this <= 6" ]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 6",
       (validation_extensions.deserialization_cel_validation) ="this <= 6" ]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 5",
       (validation_extensions.deserialization_cel_validation) = "this <= 5"]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 4",
       (validation_extensions.deserialization_cel_validation) = "this <= 5" ]}

message Foo {
  required int32 f = 1 [ 
      (validation_extensions.constructor_cel_validation) =  "this <= 4",
       (validation_extensions.deserialization_cel_validation) ="this <= 4" ]}

Thus, a message field's constructor validation must entail the deserialisation validation.

Checking whether the invariant holds, and whether a proposed evolution is legal, can prove difficult.

CEL ensures that evolutions of field validations are compatible with their previous schema versions using a lattice based type-system.

Although somewhat overkill, I believe that Z3 would be sufficient to check the invariant, and to check that the constructor validation entails the deserialization validations of all previous schema versions.

AI usage disclosure.

All prose was handwritten, with predictive autocomplete disabled. Diagrams were generated. Proof-read by Claude Opus 4.5 on DATE.

Use the word "schema evolution"