Schema Evolution over Time

Before we enter the practical part of this book, we need to touch on the last important bit about Protobuf. Often, the data schemas evolve and it is important to make sure that newer versions of our app can still interact with older/newer versions of it. Now, while Protobuf does not provide any automatic way of ensuring compatibility between different versions, it provides constructs that can help us.

In this chapter, we’re going to cover the following main topics:

• Backward and forward compatibility
• Reserved tags and names
• How to evolve schemas safely

By the end of this chapter, you will understand how to design schemas that are backward and forward compatible, and that are safe to use across versions of your app.

Technical requirements

All the code examples that you will see in this chapter can be found in the chapter6 directory in this book’s GitHub repository (https://github.com/PacktPublishing/Protocol-Buffers-Handbook).

Backward and forward compatibility

Backward compatibility is a design that is compatible with older versions of itself. Similarly, forward compatibility is a design that is compatible with newer versions of itself. While this is simple, let’s see an example to reinforce the idea.

Backward compatibility

Let’s suppose that we have the following schema (proto/v1/id.proto):

syntax = "proto3";
message Id {
  uint32 value = 1;
}

Previously, this message was doing its job. But after monitoring our use of its values, we noticed that we are getting close to the limit of a uint32 (4,294,967,295). We now need to update the type of value so that it includes more values. But we also need to make sure that previous messages with a uint32 ID are still handled properly.

Let’s see what this means by creating a new version of our schema (proto/v2/id.proto):

syntax = "proto3";
message Id {
  uint64 value = 1;
}

Now, we can...

Disabling tags – reserved tags

Protobuf has a concept called reserved tags. This means that these tags are made unusable by developers when they are updating a message. This looks like this:

message Id {
  reserved 1;
}

In this case, tag 1 isn’t reusable. This might not be directly clear how, but this helps with the problems that we saw in the previous section. Let’s see how.

If you recall the problem of integer overflow, we had a value of 4,294,967,297 encoded and Protobuf, after decoding, returned a value equal to 1. This problem came from the fact that we changed the type of the value field from uint32 to uint64 and we are now trying to encode an uint64 in an uint32. While this did not crash and the value field was populated with data, in most cases, we won’t want the overflow behavior. This might lead to getting the wrong data for a user, overwriting data that is not from the user, and more.

To prevent this, instead of directly changing...

Disabling field names – reserved names

Important message

For the sake of simplicity, I will be using Go for this example. If you are not familiar with this language, don’t worry. The example is small, and Go is a simple language. I am 100% confident that you can understand what’s going on.

We previously saw the need for reserved tags and how they can help with backward and forward compatibility. In this section, we will focus on the impact of the schema’s evolution on the code we are writing.

As we know, Protoc is mostly used to generate code out of proto files. This generated code is then used in our application to abstract the Protobuf encoding/decoding. While this is great, this creates some problems when the schemas evolve. Some of these problems can only be solved by going through the code manually and updating it but others are preventable.

Let’s see an example of a preventable problem. As you may recall, we now have v1, v2, and...

The evolution rules

We saw multiple problems caused by the evolution of the proto file and it is important to keep these in mind. So, it is time to recapitulate and formulate rules on how to evolve schemas in Protobuf.

Here’s the list of rules that we should follow if we need to maintain backward and forward compatibility:

• Never remove a reserved statement. This probably goes without saying but reserved statements are here to protect us. Do not remove or modify them; otherwise, you will get undefined behaviors.
• Never modify a field tag. This will lead Protobuf to decode data into unknown fields between versions. This means you will not receive that data and get a default value.
• Avoid reusing a field tag. As we saw, this can cause problems such as integer overflow or other problems due to type conversion. It is better to just add a new field to deal with the new data. Remember that tags are encoded as varints.
• Always add a reserved tag when deleting a field...

Summary

In this chapter, we saw the problems that might be created by evolving our schemas and how to solve each of them. We also derived a set of rules that we can follow when we are in the process of such evolution. This concludes the “theoretical” part of this book, which means we can confidently go to the practical part.

In the next chapter, we are going to create a CLI application in Go for an address book where the data is serialized and deserialized with Protobuf. This will help us practice the Protobuf syntax, the use of protoc, and the use of its generated code.

Quiz

Answer the following questions to test your knowledge of this chapter:

1. What is backward compatibility?
   1. The ability to interact with newer versions of our app
   2. The ability to interact with older versions of our app
2. What is forward compatibility?
   1. The ability to interact with newer versions of our app
   2. The ability to interact with older versions of our app
3. Would there be any backward compatibility problem if, instead of changing the field type from uint32 to uint64, we were changing from uint64 to uint32?
   1. Yes, we might have an overflow
   2. No, all uint32 values are included in uint64
4. Would there be any forward compatibility problem if, instead of changing the field type from uint32 to uint64, we were changing from uint64 to uint32?
   1. Yes, we might have an overflow
   2. No, all uint32 values are included in uint64
5. What could potentially happen if we changed a field type from int32 to uint64 (hint: think in both forward and backward compatibility terms)?
   1. Negative values will be encoded...

Answers

Here are the answers to this chapter’s questions:

1. B
2. A
3. B
4. A
5. C
6. C
7. B and/or C