The Modern Data Stack

Data’s exponential growth and the need to incorporate analytics-driven decision-making have brought new challenges and requirements to organizations. Information must be reliable, scalable, and delivered quickly to remain competitive. In this scenario, a new set of tools, technologies, and processes starts to gain traction – the Modern Data Stack (MDS).

In this chapter, you will learn what the MDS is, the principles that differentiate it from legacy stacks, and its advantages and disadvantages. Although this set of tools brought significant technological advancements versus legacy tightly coupled systems, it is important to be aware of its pitfalls and the recent considerations when choosing this type of stack.

The main topics that are going to be covered are as follows:

• Understanding the Modern Data Stack
• Explaining three key differentiators versus legacy stacks
• Discussing the advantages and disadvantages of the MDS

...

Understanding a Modern Data Stack

As the name suggests, the MDS represents a technological evolution compared to previous systems widely used in recent decades. From the development of the business data warehouse in the 1980s to the rise of cloud technology with Amazon Web Services (AWS) in the early 2000s, on-premises legacy data stacks dominated the landscape. These systems had a monolithic IT infrastructure, resulting in complex maintenance. The MDS transformed this scenario – bringing modularity and cloud-native tools. However, before we dive into the details, let’s first define what a data stack is.

A data stack is a collection of tools and services as part of an extensive technology infrastructure designed to ingest, store, transform, and serve data. It makes data accessible across an organization and is fundamental to delivering business insights through reporting and dashboards, advanced analytics, and Machine Learning (ML) applications. Figure 2.1 illustrates...

Explaining three key differentiators versus legacy stacks

The MDS brought a series of technological advancements in comparison with precursor systems (pre-cloud). As demonstrated in Figure 2.1, the main differentiators are its SQL-first approach, cloud-native tools, and managed and modular solutions. In this section, we are going to deep dive into the details of these characteristics.

Lowering technical barriers with a SQL-first approach

Developed in the 1970s by IBM researchers, SQL – or SEQUEL as it was called originally – was designed to access data in an integrated relational database, by both professional programmers and more occasional database users. Here is a link to the full version of the paper, published by Donald D. Chamberlin and Raymond F. Boyce (1976): https://doi.org/10.1145/800296.811515.

Decades later, SQL remains indispensable for this purpose, and its popularity has increased sharply. In the Stack Overflow Developer Survey 2022, it was the...

Discussing the advantages and disadvantages of the MDS

Although the MDS simplifies the data life cycle significantly, it is important to highlight and discuss its pitfalls.

The advantages can be summarized as follows:

• Speed of delivery: Setting up an MDS can take as little as a few hours to a couple of days, depending on the specific use case. Additionally, its self-service characteristic brings data products and solutions closer to where the business questions arise.
• Cost: As the MDS is cloud-based and uses out-of-the-box tools, costs are cut down significantly due to reduced complexity in architecture and hardware. In addition, cloud-native data warehouses provide cheaper data processing and storage when compared to on-premises systems.
• Democratization: Here, we see two dimensions of democratization – lowered technical barriers and the dissemination of information across an organization. The combination of managed ingestion and SQL-first tools lowers the...

Summary

In this chapter, you learned what the Modern Data Stack is and how it can transform the work of data teams by lowering technical barriers, improving the speed of information delivery, and bringing modular and managed solutions. With its combination of a SQL-first approach, cloud-based systems, and managed and modular tools, the MDS brings improvements compared to legacy, on-premises, and monolithic stacks. Still, it is important to be aware of its pitfalls, such as the large number of tools and considerations about siloed information.

With the MDS, analytics engineers can leverage managed ingestion and data transformation tools to deliver end-to-end analytics.

In the following chapter, you will get the opportunity to deepen your knowledge of one of the MDS’s components – data ingestion. You will learn common approaches to it and what the steps are to set up an efficient and performant ingestion pipeline.