Exploring the types of scaling
When we talk about scaling in the context of ETL pipelines, it essentially means enhancing the pipeline to handle more data. There are two primary ways to achieve this – vertical scaling (scaling up) and horizontal scaling (scaling out).
Vertical scaling
Vertical scaling involves enhancing the capabilities of the existing system. It’s called “scaling up” since it works by increasing the processing power of the current machine or node (i.e., in a Kubernetes cluster). This can range anything from simply updating the software of the machine to optimization configurations to even adding parallel processing within the same resource. Vertical scaling can be accomplished using physical hardware or cloud resources.
Figure 12.5: Visual representation of vertical scaling
The benefit of vertical scaling is that it’s often straightforward to implement – you add more resources, and your...
