CI/CD with GitHub

In this chapter, we will introduce Continuous Integration/Continuous Delivery (CI/CD) and how to apply CI/CD in your Scala data engineering project using GitHub. CI/CD is a set of best practices and tools that automate the development, testing, and deployment of data pipelines and workflows. It involves the continuous integration of code changes, automated testing, version control, and continuous delivery of pipeline deployments. The goal of CI/CD is to streamline the development process by enabling rapid iteration, reducing errors, and ensuring consistent quality.

We will cover the following main topics:

• Introducing CI/CD and GitHub
• Working with GitHub
• Deploying pipelines
• Understanding GitHub Actions

Technical requirements

Install the following command-line tools:

• Git CLI: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git
• GitHub CLI: https://cli.github.com/

Introducing CI/CD and GitHub

One of the most common platforms for implementing CI/CD and collaborating on code is GitHub. GitHub and its ecosystem provide a variety of tools to manage your data engineering project from development to deployment to production.

In this section, let’s cover how CI and CD work on GitHub.

Understanding Continuous Integration (CI)

The core concept of CI on GitHub revolves around continuously integrating code changes from multiple contributors into a shared repository. Whenever developers push changes to the repository, automated build and test processes are triggered. This enables teams to detect integration issues, conflicts, and errors early on, promoting a more stable and coherent code base. Through the use of GitHub Actions or third-party CI tools such as Jenkins or CircleCI, developers can define custom workflows to automate the build and test stages, ensuring code integrity. We will cover GitHub Actions later in this chapter.

Understanding...

Working with GitHub

There are many different ways to interact with GitHub. You can use an IDE’s interface or extensions, a web user interface, or a command-line interface (CLI). We’ll focus on the Git and GitHub CLIs for this portion of the book.

The Git CLI, or git, is the command-line tool that interacts directly with the Git version control system. It provides a set of commands to manage the entire lifecycle of a Git repository, such as creating repositories, initializing a new repository, cloning repositories, managing branches, committing changes, merging branches, and more. It is installed on your local machine and can be used with any Git-compliant repository.

The GitHub CLI, or gh, is a command-line tool provided by GitHub that extends Git’s functionality with additional features and allows you to interact with GitHub repositories directly from the command line. It also offers commands for creating repositories; cloning repositories; managing pull...

Understanding GitHub Actions

GitHub provides a wide range of tools and features to support CI/CD workflows. It offers a powerful marketplace of pre-built CI/CD actions, allowing developers to easily integrate popular tools and services into their workflows. GitHub Actions also enables the creation of custom workflows using YAML-based configuration files, providing flexibility and control over the entire CI/CD pipeline. By leveraging these tools, developers can automate code formatting, run unit tests, perform code reviews, and deploy applications seamlessly.

The following sections introduce the components of GitHub Actions with a running example.

Workflows

Workflows are the top-level objects under GitHub Actions. They provide a logical grouping of all of the jobs that you plan to run as part of the CI/CD pipeline and have associated trigger events. These are defined in the .gihub/workflows/ directory in your project root folder. A typical workflow looks like the following...

Summary

In this chapter, we learned how to take our code changes and manage them in a collaborative environment while maintaining the code quality and health of our code base. We did this by learning how to work with GitHub by committing and merging code using the Git and GitHub CLIs. We also learned how to use GitHub Actions to promote code from development to production.

In the next chapter, we are going to look at the orchestration of data engineering pipelines through various open source tools.