Automating Workflows

In the previous chapter, you got acquainted with challenges surrounding ownership and accountability in a development team, ensuring coding standards are adhered to and task tracking. We had a look at how to do a code review and got to know some best practices around writing documentation for easily onboarding new colleagues. We briefly mentioned Continuous Integration/Continuous Deployment or CI/CD as a way to do functionality tests and formatting checks on your code and deploy changes into production in an automated fashion. In this chapter, you will learn about data orchestration and automating workflows, and we will give you all the details you need to know to successfully build a CI/CD pipeline as an analytics engineer.

By the end of this chapter, you will have gained a comprehensive understanding of several key concepts and practices. Firstly, you will understand the critical role of CI/CD within the realm of DataOps. Additionally, you will have a clear...

Introducing DataOps

DataOps is a relatively new term and is derived from the more prominent term DevOps, which has been around for quite some time now. This term hails from software engineering and refers to the practices that aim to automate and streamline the software delivery process, making it faster, more reliable, and more efficient.

DevOps and DataOps are related. DevOps is a set of practices that emphasizes collaboration and communication between development and operations teams. The goal of DevOps is to streamline the software development and deployment process, enabling teams to release software faster and more reliably.

The key elements of DevOps include continuous integration (CI), continuous delivery, and continuous deployment (CD). CI involves regularly merging code changes into a central repository and testing this code, while CD involves automating the deployment of code changes.

With the increasing importance of data in today’s world, a new term has...

Orchestrating data pipelines

Data orchestration refers to the automated and coordinated management of data workflows across various stages of the analytics life cycle. This process involves integrating, cleansing, transforming, and moving data from diverse sources to a data warehouse, where it can be readily accessed and analyzed for insights.

We have already covered how dbt empowers data engineers and data analysts to move away from repetitive stored procedures and employ version control and testing found in traditional software engineering to help run data workflows. dbt is largely used for the transformation part of the analytics workflow.

On the other hand, the goal of data orchestration in analytics engineering is to streamline the flow of data through different tools and processes, ensuring that it is accurate, timely, and in the right format for analysis. This involves scheduling dbt jobs in the correct sequence – for example, managing dependencies between tasks...

Continuous integration

CI is part of CI/CD, a practice hailing from software engineering and DevOps. CI allows developers to test their code before code is merged into production, in an automated fashion.

Integration

The term integration in CI refers to the process of combining code changes from multiple contributors into a shared unified code base. The key principle behind CI is to automate the process of merging code changes regularly and consistently, thereby reducing the risk of merge conflicts that may arise when integrating changes made by different team members.

Merge conflicts arise when code from different branches is merged, and certain code or functionality is present in one branch but not in the other, where you may have different lines of code. When you build on a separate feature branch, the code will probably have diverged from what is in the main branch.

Before code is merged to production, this could be your main or master branch in your version control...

Continuous deployment

In analytics engineering, CD refers to the automated process of deploying all changes in data models, scripts, and configurations to the production environment, post-testing. This practice ensures that new features, bug fixes, and updates are swiftly and reliably deployed into production, usually on the condition that the CI pipeline has been completed successfully. Therefore, in essence, a deployment job deploys the modified models into production. As you can imagine, this defaults to running a dbt build or run command, just like any other dbt job. The subtle difference is in the naming – a deployment job is a dedicated job that is only ever called after a CI job.

Let’s break it down:

• Deployment: The deployment aspect of CI/CD refers to deploying changes in your dbt models, YAML configuration files, and any changes you have committed to your development feature branch. Essentially, with deployment, we mean running your dbt models in a...

Continuous delivery

Continuous delivery also involves testing any code change, whether it’s a new feature, bug fix, or experiment, and preparing it for release to a production environment. However, the actual deployment to production in continuous delivery is a manual step, requiring a human decision to make the final push. The goal of continuous delivery is to ensure that software can be reliably released at any time, creating a stable, reliable, and repeatable process for releasing software. The benefits include having a deployable product after every change and more reliable software delivery. This approach also eases the pressure on the decision to release as the product is always ready for deployment.

First, let’s understand how continuous delivery and CD are different.

Continuous delivery versus continuous deployment

Continuous deployment (CD) and continuous delivery, while closely related in the realm of software development, particularly in automated...

Summary

In this chapter, we saw how integrating and deploying your code can be automated. We also discussed how analytics engineering is leveraging existing practices from software engineering when it comes to using CI/CD for automated tests and deployment to ensure code quality and delivery promptly. Testing code for formatting, compilation errors, and unit testing functions and macros is an essential part of testing your code before deploying to production.

In this chapter, we introduced the paradigm of DataOps, its ideas and principles, and how it is strongly influenced by practices from software engineering. Then, we introduced the concept of CI, discussed the necessity of testing your code for formatting issues, compilation and runtime errors, and covered unit testing your functions and macros. Next, we talked about CD, the concept of state comparison and idempotency, and enabling Slim CI/CD runs in dbt Cloud using state deferral to reduce costs and time. Lastly, we briefly...