Transforming Data

In the fast-paced world of data-driven decision-making, the ability to transform raw data into valuable insights is a critical competitive advantage. This is part of the foundation of analytics work: enabling organizations to extract meaningful information from vast and complex datasets.

In Chapter 5, Data Modeling, we learned about the benefits and techniques of data modeling. A well-designed data model guides the transformation process that turns raw data into ready-to-use data.

This chapter covers the following topics:

• Transforming data – the foundation of analytics work
• Design choices
• Tools that facilitate data transformations

Transforming data – the foundation of analytics work

Raw data is of little value unless it is transformed into valuable insights. In this section, we look into why transforming data is such a crucial step within the data value chain.

A key step in the data value chain

The digitalization of businesses, rapid advancements in technologies such as the Internet of Things (IoT), and the proliferation of smart devices, social media, and advertising platforms have generated abundant data that companies can leverage to understand their customers better, optimize their operations, and enhance their products and services. These phenomena, together known as big data, have revolutionized how organizations operate, make decisions, and generate revenue.

However, collecting data is not enough to create a competitive edge in the market. The zettabytes of data that organizations collect daily must be transformed into actionable insight to be of any value. Quantity does not equal quality...

Design choices

To implement data transformations that are robust and scalable, we must make some conscious design choices. Agreeing on how and where you will transform your data will allow your team to collaborate more effectively and coherently across pipelines.

Where to apply transformations

As seen in Chapter 2, The Modern Data Stack, a high-level architecture of the data stack resembles the following:

Figure 6.2 – High-level architecture example of a data stack (see Chapter 2, The Modern Data Stack)

At each of these steps, you might consider applying transformations. For instance, as seen in Chapter 3, Data Ingestion, transformations performed during ingestion focus on shaping data into a format and structure that are compatible with the destination system, such as a relational database. This mainly involves parsing and translating source data. Sometimes, however, one might consider cleaning, aggregation, and enrichment during ingestion...

Data transformation best practices

As seen in previous chapters, analytics engineering embraces software engineering best practices to model, transform, test, deploy, and document data in a reusable way.

When it comes to writing transformation pipelines, SQL is the industry standard. Still, you might also want to use other languages, such as Python or Scala, depending on the tools you use for transformation.

The barrier to entry to writing SQL code is quite low. Thanks to its declarative nature, SQL is easy to read. Most data specialists know how to write SQL, making it easier for organizations to hire talent who can work with SQL pipelines, an important factor in democratizing transformation capabilities.

In this section, we will tackle SQL best practices for your transformation pipelines. We will also mention language specific to dbt and Databricks. In dbt, the SQL files in which developers write SELECT statements are called models. In Databricks, code is organized within...

Tools that facilitate data transformations

As data volumes, complexity, and usage continue to grow, tools that facilitate data transformations have emerged to simplify and streamline this process. During the early 2020s, dbt became one of the most prominent and used tools for data transformation.

These tools have revolutionized the way data engineers and analysts work, making it easier and more efficient to transform and analyze data. Previously, data transformations consisted of a series of files containing SQL queries, requiring extensive coding and scripting to ensure they ran in the desired order.

Now, most tools can transform data, sometimes with low or even no code. However, the fact that you can doesn’t mean that you should. Most of the time, the answer to whether you should transform data in a tool is “it depends.” The following section explores what to consider when making that decision.

Types of transformation tools

As seen in the Where to...

Summary

In this chapter, we have discussed the importance of data transformation pipelines as a crucial step in the data value chain. We have delved into the practical topics and best practices that developers should consider when designing their transformation pipelines.

Finally, we explored the families of tools that facilitate transformation and how to break down your needs to identify the right tools for your organization in an evolving industry.

In the next chapter, we will delve into the next step of the analytics workflow: Serving data.