CI/CD best practices in data engineering
After learning how to use the GCP tools to implement CI/CD, you now hopefully have an idea of the possibilities available to a data engineering team when implementing CI/CD on a data platform.
You may start to think of some ideas, such as the following:
- Checking how clean your code is
- Checking your data quality before going into productions
- Planning automatic testing in different environments
In data engineering specifically, after working with dozens of companies from many industries, I haven’t seen any golden standard on how to implement CI/CD. It depends on the skill set of the team, the number of people in the team, the complexity of the systems, and the budget.
What makes this topic exciting is that it’s still evolving. Even though there is no golden rule and almost endless possibilities, I saw some patterns. In this section, I will share my thoughts on the considerations and best practices I&...
