Delving into the internals of Storm

Now that we know which physical components are present in a Storm cluster, let's understand what happens inside various Storm components when a topology is submitted. When we say topology submission, it means that we have submitted a distributed job to Storm Nimbus for execution over the cluster of supervisors. In this section, we will explain the various steps that are executed in various Storm components when a Storm topology is executed:

• Topology is submitted on the Nimbus node.

• Nimbus uploads the code jars on all the supervisors and instructs the supervisors to launch workers as per the NumWorker configuration or the TOPOLOGY_WORKERS configuration defined in Storm.

• During the same duration all the Storm nodes (Nimbus and Supervisors) constantly co-ordinate with the Zookeeper clusters to maintain a log of workers and their activities.

As per the following figure, we have depicted the topology and distribution of the topology components, which are the same across clusters:

In our case, let's assume that our cluster constitutes of one Nimbus node, three Zookeepers in a Zookeeper cluster, and one supervisor node.

By default, we have four slots allocated to each supervisor, so four workers would be launched per Storm supervisor node unless the configuration is tweaked.

Let's assume that the depicted topology is allocated four workers, and it has two bolts each with a parallelism of two and one spout with a parallelism of four. So in total, we have eight tasks to be distributed across four workers.

So this is how the topology would be executed: two workers on each supervisor and two executors within each worker, as shown in the following figure: