Chapter 7. Advanced Load Balancing and Circuit Breakers

In this chapter, we will continue the subject discussed in the previous chapter, inter-service communication. We will extend it to more advanced samples of load balancing, timeouts, and circuit breaking. 

Spring Cloud provides features that make implementation of communication between microservices nice and simple. However, we must not forget that the major difficulties we would face with such communication concern the processing time of the systems involved. If you have many microservices in your system, one of the first issues you need to deal with is the problem of latency. In this chapter, I would like to discuss a few Spring Cloud features that help us to avoid latency problems that are caused by many hops between services when processing a single input request, slow responses from several services, or a temporary unavailability of services. There are several strategies for dealing with partial failures. These include setting network...

Load balancing rules

Spring Cloud Netflix provides different load balancing algorithms in order to provide different benefits to the user. Your choice of supported method depends on your needs. In the Netflix OSS nomenclature, this algorithm is called a rule. The custom rule class should have implemented an IRule base interface. The following implementations are available by default inside Spring Cloud:

• RoundRobinRule: This rule simply chooses servers using the well-known round robin algorithm, where incoming requests are distributed across all instances sequentially. It is often used as the default rule or fallbacks for more advanced rules, such as ClientConfigEnabledRoundRobinRule and ZoneAvoidanceRule. ZoneAvoidanceRule is the default rule for Ribbon clients.
• AvailabilityFilteringRule: This rule will skip servers that are marked as circuit tripped or with a high number of concurrent connections. It also uses RoundRobinRule as a base class. By default, an instance is circuit tripped if an...

Customizing the Ribbon client

Several configuration settings of the Ribbon client may be overridden with Spring bean declarations. As with Feign, it should be declared in the client annotation field named configuration, for example,@RibbonClient(name = "account-service", configuration = RibbonConfiguration.class). The following features may be customized with this approach:

• IClientConfig: The default implementation of this is DefaultClientConfigImpl.
• IRule: This component is used to determine which service instance should be selected from a list. The ZoneAvoidanceRule implementation class is auto-configured.
• IPing: This is a component that runs in the background. It is responsible for ensuring that the instances of service are running.
• ServerList<Server>: This can be static or dynamic. If it is dynamic (as used by DynamicServerListLoadBalancer), a background thread will refresh and filter the list at a predefined interval. By default, Ribbon uses a static list of servers taken from configuration...

The circuit breaker pattern with Hystrix

We have already discussed the different implementations of load balancer algorithms in Spring Cloud Netflix. Some of them are based on monitoring the instance response time or the number of failures. In these cases, a load balancer makes decisions about which instance should be invoked based on these statistics. The circuit breaker pattern should be treated as an extension of that solution. The main idea behind a circuit breaker is very simple. A protected function call is wrapped in a circuit breaker object, which is responsible for monitoring a number of failure calls. If the failures reach a threshold, the circuit is opened, and all further calls will be failed automatically. Usually, it is also desirable to have some kind of monitor alert if a circuit breaker trips. Some crucial benefits derived from the usage of the circuit breaker pattern in your applications are the ability to continue operating when a related service fails, the prevention...

Monitoring latency and fault tolerance

As I have already mentioned, Hystrix is not only a simple tool implementing a circuit breaker pattern. It is a solution that deals with latency and fault tolerance in distributed systems. One interesting feature provided by Hystrix is the ability to expose the most important metrics related to interservice communication and display them using a UI dashboard. This function is available for clients wrapped with the Hystrix command. In some previous samples, we have analyzed only a part of our system to simulate a delay in communication between customer-service and account-service. That's a really good approach when testing advanced load balancing algorithms or different circuit breaker configuration settings, but now we will go back to analyzing the whole of our sample system setup as a set of standalone Spring Boot applications. This allows us to observe how Spring Cloud, in conjunction with Netflix OSS tools, helps us to monitor and react to latency...

Failures and the circuit breaker pattern with Feign

The Feign client is, by default, integrated with Ribbon and Hystrix. This means that, if you wish, you can apply different approaches to deal with latency and timeouts in your system when using that library. The first of these approaches is a connection retry mechanism provided by the Ribbon client. The second is a circuit breaker pattern and a fallback implementation available under the Hystrix project, which has already been discussed in the previous sections of this chapter.

Retrying the connection with Ribbon

Hystrix is enabled by default for the application when using a Feign library. This means that you should disable it in the configuration settings if you do not want to use it. For the purpose of testing a retry mechanism with Ribbon, I suggest that you disable Hystrix. In order to enable connection retrying for Feign, you only have to set two configuration properties—MaxAutoRetries and MaxAutoRetriesNextServer. The important settings...

Summary

You may not be aware of the configuration settings or tools described in this chapter if you have already been using auto-configured clients for inter-service communication. However, I think that it is worth having some knowledge about a few of the advanced mechanisms, even if they can run in the background and/or out of the box. In this chapter, I have tried to give you a closer view on topics, such as load balancers, retries, fallbacks, or circuit breakers by demonstrating how they work using simple examples. After reading this chapter, you should be able to customize Ribbon, Hystrix, or Feign clients to suit your needs related to communication between microservices, both on a small and large scale. You should also understand the when and why of using them in your system. With this chapter, we are closing the discussion about the core elements inside microservices-based architecture. Now, we have got one more important component to look at that is outside the system by quite a...