Chapter 4: Data Center (Not Trade) Federation

I'm very happy to tell you (and myself) that after that very challenging chapter dealing with Consul security aspects, our discussion about extending your Consul cluster will be much lighter. In fact, as many of the features we'll be discussing in this section are specific to Consul Enterprise, there won't be any exercises!

In the previous chapters, we focused primarily on setting up and securing our Consul clusters. However, as Consul is implemented in enterprise environments, we want to look at how to scale the Consul system itself and extend that Consul functionality across multiple data centers or regions. To understand the expansion of Consul, we'll be reviewing the following topics:

• Scales – Nope, we won't be discussing reptilian or musical scales, but rather how to scale the Consul system. We've seen in earlier chapters how increasing server clusters can negatively impact the time...

Protection and scale

Although Consul agents can work with services on external nodes, best practices (and best flexibility) recommend having a Consul agent co-resident on the nodes running the connected services. In many enterprises, Consul is connecting thousands of services (or more), which means Consul clusters of thousands, or tens of thousands, of nodes. Certainly, we don't want only three to five server nodes managing a cluster of that size, so let's start with expanding the Consul server cluster.

As we learned in Chapter 2, Architecture – How Does It Work?, bigger is not always better when it comes to the size of your consensus pool. We need at least three servers to obtain a quorum, but as that server count increases, it's going to take much longer for the pool to elect a leader. In order to resolve this, Consul offers the ability to configure Consul server agents as a read replica. When set with this flag, the agent doesn't participate in any...

Subdivisions

Utilizing the Consul namespace feature, we are able to effectively subdivide our entire Consul system into multiple mini-systems, each with its own autonomy. This offers multiple benefits to the greater enterprise organization:

• It utilizes a single centralized team to operate and manage Consul at a corporate scale. This not only provides consistency with how services are managed but also reduces the overhead of managing multiple independent Consul systems.
• It enables individual teams to manage and secure their own applications without relying on multilayer management and control. Although the centralized operators of Consul have full control, autonomy can be granted to teams (within bounds) to manage their services as they see fit.
• Splitting up Consul not only subdivides the operations of the system but also the service catalogs themselves. This enables organizations to create services that are completely isolated from other aspects of the Consul catalog...

Connecting clusters

To think that all of our services are going to reside within a single data center might have worked 20 years ago when I started writing this book (so it seems). The world has truly evolved to a continuously connected mobile environment, although some might argue that this was actually devolution. Regardless, the only way to really maintain the communications among our services in this world is to ensure that their availability and location can be discovered from multiple geographic areas. Whether I'm in Boston, Berlin, or Melbourne, the applications and services I've grown to rely on must be available! All that we have done within this chapter has led us to this point!

Throughout the book, we've been talking a lot about gossiping and pools, and we're going to continue that discussion. However, as you can imagine, the amount of data that we have communicated within a single Consul cluster can be quite extensive, especially as we onboard all...

Summary

Well, I truly hope you enjoyed that chapter. We've seen how Consul is able to, in many real-world scenarios, scale to incredibly large numbers of nodes. We've seen how read replicas and, more importantly, the use of Autopilot enables us to scale our server clusters to support networks with tens of thousands of nodes. I believe I've even heard recently of a company running Consul in production with over 100,000 nodes! Centralizing the operation for all services on a system that large can be challenging for sure, so enabling namespaces not only helps scale the application usage within Consul but also the dependency on humans to operate the system as it grows. Finally, we've looked at federating our Consul clusters in a couple of ways to help span our service availability across data centers. As I've said since the beginning, Consul enables communication across components and services. Now those communication paths are no longer bound by scale or geography...