Introducing CI/CD and GitHub
One of the most common platforms for implementing CI/CD and collaborating on code is GitHub. GitHub and its ecosystem provide a variety of tools to manage your data engineering project from development to deployment to production.
In this section, let’s cover how CI and CD work on GitHub.
Understanding Continuous Integration (CI)
The core concept of CI on GitHub revolves around continuously integrating code changes from multiple contributors into a shared repository. Whenever developers push changes to the repository, automated build and test processes are triggered. This enables teams to detect integration issues, conflicts, and errors early on, promoting a more stable and coherent code base. Through the use of GitHub Actions or third-party CI tools such as Jenkins or CircleCI, developers can define custom workflows to automate the build and test stages, ensuring code integrity. We will cover GitHub Actions later in this chapter.
