Developing a Protoc Plugin in Golang

In Chapter 7, we created an address book in Golang. In this chapter, we are going to iterate on the AddressBook and create a protoc plugin that generates Go code to validate the phone numbers and emails that the user inputs.

In this chapter, we are going to cover the following main topics:

• Defining Protobuf custom options
• Writing a custom protoc plugin

By the end of the chapter, you will be able to understand, at both theoretical and practical levels, what custom options and protoc plugins are. More importantly, you will be able to create them by yourself in Golang.

Technical requirements

All the code that you will see in this section can be found in the directory called chapter9 in the GitHub repository (https://github.com/PacktPublishing/Protocol-Buffers-Handbook).

The project

In this chapter, we are going to create a protoc plugin that generates code validating the user input for a phone number and email address in our AddressBook application. This involves creating a Protobuf custom option and writing the actual plugin logic to generate the validation code.

The overall goal of this chapter is to have a CLI that checks the user input. Let’s say the user enters the following command:

$ go run main.go add --kind per --name Clement --phone 111

It should return the following error:

error: 111 is not a valid phone number

Similarly, for emails, let’s say the user enters the following:

$ go run main.go add --kind per --name Clement --email 111

It should return the following error:

error: 111 is not a valid email

And obviously, this should also work for company contacts.

On top of that, we will add an option to the protoc plugin that lets us choose which regexp rule to use in order to check the phone number. The...

What are custom options?

Protobuf custom options are a way to annotate part of the schema with contextual information. Think of them as normal Protobuf options but with custom names. It is as simple as that. The reason they exist is also simple – it is extensibility. If you do not find the right option for your use case, just create one by yourself.

Now, let us face it, the first time we hear about Protobuf custom options, it is hard to understand why, except for the pretty abstract extensibility concept, we would need them. As such, I want to show you some examples of custom options and how they are used.

The first project that is very popular and uses Protobuf custom options is protovalidate (https://github.com/bufbuild/protovalidate/). If you have never heard of it, it is a “series of libraries designed to validate Protobuf messages at runtime based on user-defined validation rules”.

Let us see an example. Let us say that we want the name of the user...

What are protoc plugins?

Once again, before diving into the actual code of the plugin, let us try to understand what protoc plugins are and what they do.

As we saw in the section on custom options, protoc plugins and custom options are generally used together. The custom options provide contextual information, and the plugin gets that information and acts on it. This action, more often than not, is a generation of some sort of code. We have already discussed protovalidate and gRPC-Gateway generating some code that can be used in your business logic.

Even though, in this section, we are going to generate code, it is important to understand that protoc plugins can do other things than generate code.

You can also, for example, generate documentation following the OpenAPI specification with the protoc-gen-openapiv2 provided by gRPC-Gateway. Let’s say you have the following schema:

syntax = "proto3";
import "google/api/annotations.proto";
import "...

The plugin

Important message

This section will use Golang. If you are not confident about using Golang, I would still encourage you to try and get a sense of what is going on. The best alternative to write such a plugin is C++, although it is a less beginner-friendly language.

Let us start creating our own protoc plugin. Since the protogen API (https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen) is easy to use, we are going to build it in Golang. Similarly to the work we did in Chapter 7, we are going to separate the business logic (pkg) and the CLI (cmd) part of our application. So, we will have the following file structure:

.
├─ cmd
│ └─ protoc-gen-check
│    └─ ...
├─ go.mod
├─ pkg
│ └─ protoc-gen-check
│    └─ ...
└─ proto
   └─ validate
  ...

Updating the AddressBook

As we are going to work with the application we wrote in Chapter 7, let’s copy it over to the chapter9 directory:

$ cp -R ../chapter7/cmd/addressbook cmd
$ cp -R ../chapter7/pkg/addressbook pkg
$ cp ../chapter7/main.go .
$ cp ../chapter7/proto/addressbook.proto proto

Now, because we were using a Go module name including chapter7, we will need to replace all the occurrences by chapter9. On Linux/Mac, we can run the following command:

$ find . -type f -exec sh -c "sed -i '' -e 's/chapter7/chapter9/g' {}" ";"

And on Windows (Powershell), we can run the following:

$ Get-ChildItem -Recurse -File -Include *.proto,*.go,go.mod | ForEach { (Get-Content $_ | ForEach  { $_ -replace ‘chapter7’, ‘chapter9’ }) | Set-Content $_ }

We are now all set up to start.

The first step is to update our proto file to use our custom option. To do that, we will import our validate.proto...

Adding a protoc plugin option

You might already have an idea of how to add options to the plugin. If you remember, we created an instance of protogen.Options in our main plugin. This Options object can work with the flags package in the standard library. We simply need to define our flags and pass the Set function to the Options instance. This looks like this (cmd/protoc-gen-check/main.go):

import (
  "flags"
  //...
)
func main() {
  var flags flag.FlagSet
  phoneRegexp := flags.String("phone_regexp", "", "custom regex for 
  phone checking.")
  opts := protogen.Options{
    ParamFunc: flags.Set,
  }
  //...
}

If you are not familiar with the flags package from the standard library, we define a flag of type string, the name phone_regexp, and the default value of the empty string.

We can now pass the phoneRegexp variable to our generateFile...

Summary

In this chapter, we looked at what Protobuf custom options and protoc plugins are. We saw that they work in pairs to achieve some kind of generation (code, documentation, etc.). We then saw how to develop a plugin in Golang and how to make it configurable. Finally, we applied our knowledge of the AddressBook project to validate user input.

In the next chapter, we will go over the different ways to build projects that use Protobuf. We will talk about how to build a project manually, with Buf and Bazel.

Challenges

• Implement the same plugin that generates Python code and updates the Python AddressBook
• Implement a plugin (protoc-gen-redact) that generates a Redact function, implementing the redaction of data labeled with the debug_redact field option
• Add a protoc plugin option to choose the character that should be used for redacting (the default is *)