Advanced Build

In Chapter 4, we talked about the Protobuf compiler (protoc). While this is the way in which we built our proto files up until now to learn about Protobuf, there are other build tools that are widely used in the industry, and it is important to be aware of them. In this chapter, we are going to cover two other ways to build Protobuf projects. We are first going to recapitulate what we have learned about protoc, and then we will see how to use Buf and Bazel.

In this chapter, we are going to cover the following main topics:

• Building with Makefile
• Building with Buf
• Building with Bazel

By the end of the chapter, you will be able to understand, at both theoretical and practical levels, what custom options and the protoc plugins are. More importantly, you will be able to create them by yourself in Golang.

Technical requirements

All the code that you will see in this section can be found in the directory called chapter10 in the GitHub repository (https://github.com/PacktPublishing/Protocol-Buffers-Handbook).

The sample project

In this chapter, in order to not wrestle too much with the build system, we are going to use a smaller Go project than the AddressBook. This project, however, is designed to represent the minimal project in which a build system is valuable. From there, you should be able to, with a bit more configuration, adapt the builds we create here to your own project.

The sample project will have the following file structure:

.
├─ go.mod
├─ go.sum
├─ main.go
└─ proto
   ├─ test.proto
   └─ v1
      └─ test.proto

Notice that we are nesting proto files in the proto directory and the rest of the application is in the root directory. We decided to nest the proto files by at least one level because in this chapter we want to show how to discover multiple proto files that are stored in multiple levels of the proto directory...

Building manually with protoc – a summary

As we saw in Chapter 4, we can build proto files with protoc. On top of that, protoc is also a convenient tool for learning about the Protobuf internals because we can use --encode and --decode flags. However, as with every tool out there, protoc has some limitations.

The main limitation comes when the project starts to have a lot of proto files. Let’s say that you have 50 proto files you want to generate C++ code from. You will need to write a command that looks like the following:

$ protoc --cpp_out=. schema1.proto schema2.proto ... schema50.proto

You basically have to type every single file you want to generate code from and the corresponding output location with options. This means that you will have to write a huge command. And, obviously, nobody wants to do that manually.

That is why we have build tools that automatically do that for us. Let’s talk about the different tools available, starting with Makefiles...

Using Makefile

Important message

While it is possible to use Makefiles on Windows (e.g. choco install make), this is much more a Unix thing because GNU Make is already installed on Linux and macOS. If you control the Windows environment on which the code will be running, it is totally feasible for you to use them. However, if you expect other users to run your code, this might not be the best solution.

GNU Make (https://www.gnu.org/software/make/) is a tool that lets you automate some tasks. It has a powerful syntax in which you define rules (https://www.gnu.org/software/make/manual/html_node/Rules.html) and their behavior. And later, you execute the make $target command to run the behavior you defined. Before generating code from proto files, let’s see an example.

We will create a basic rule called hello that prints Hello, world on the standard output. Create a file called Makefile and add the following content to it:

hello:
    @echo "Hello...

Using Buf

Important message

For this section, you are going to need Buf CLI. You can find how to install it here: https://buf.build/docs/installation.

The next tool that is important to know is Buf (https://buf.build/). This is mainly a CLI tool that deals with your proto files in some way. It can build them, lint them, format them, check for breaking changes, and so on. It is very interesting because it can help you with every step of the CI/CD pipeline.

Let’s see an example of how to use the CLI. Let’s assume that we have a similar project to what we had in the previous section:

.
├─ go.mod
├─ main.go
└─ proto
   ├─ test.proto
   └─ v1
      └─ test.proto

Now, at the root of the proto files (the proto directory), run the following command:

$ buf mod init

This should create a buf.yaml file in the proto directory...

Using Bazel

Important message

This section will use Bazel. In order to download it, consider using Bazelisk since it will automatically update the Bazel version to the latest LTS release. You can find how to install Bazelisk here: https://bazel.build/install/bazelisk.

The last tool that I consider worth considering is Bazel. Unlike Makefiles and Buf, Bazel is a full-blown build system. As such, it requires more configuration, but it can build multiple languages, has efficient caching, and more. Once again, let’s try to run the sample project using this tool.

A modern Bazel project should start by adding a file called MODULE.bazel at the root of the directory. This file contains the dependencies that are needed to build our project. In our case, we will need to have utilities to build Protobuf and Go. We will also need to have the dependency on Protobuf Go and to manage that we will use Gazelle (https://github.com/bazelbuild/bazel-gazelle). Our MODULE.bazel looks like...

Summary

In this chapter, we saw three popular ways of building Protobuf projects. The first one was using some kind of automation tool, such as Makefile. The second used the highly specialized tool called Buf. And the third one was using the highly generalized build system called Bazel. All of these have some advantages and disadvantages, and our job as engineers is to decide which tool is right for the job.

This book ends with this chapter. I hope you have learned something from all the theory and the practical parts. Protobuf is a very interesting data format, and with the rise of gRPC, Protobuf has become a must-have skill. With this book, you have learned everything from the serialization internals to how to use Protobuf in Python and Golang. I hope you liked the adventure, and I hope it will help you during your career.

Challenge

• Try adding one of the build systems to the AddressBook in Golang.
• Do the same with the AddressBook in Python.
• Try to create a project in your favorite programming language and build it with one, two, or all of the build systems shown in this chapter.