Enhancing the Speed and Maintainability of CI/CD Pipelines

In this chapter, we’re going to cover the different tools and methods you can leverage to enhance the speed and maintainability of your CI/CD pipelines. Our goal in this chapter is to cover three main methods for how to speed up your pipelines. We don’t plan on covering every method to speed up a CI pipeline in GitLab, but rather the most impactful methods on pipeline speed.

We will cover the following topics in this chapter:

• Accelerating pipelines with directed acyclic graphs and parent-child architecture
• Building code for multiple architectures
• When and how to leverage caching or artifacts
• Reducing repeated configuration code with anchors and extensions
• Improving maintainability by combining multiple pipelines and leveraging parent-child pipelines
• Securing and accelerating jobs with purpose-built containers

Accelerating pipelines with directed acyclic graphs and parent-child architecture

GitLab supports the usage of the directed acyclic graph (DAG) pattern for building CI pipelines. Under normal usage with GitLab, each stage represents a series of jobs that need to be completed. Each stage is comprised of multiple jobs that are executed in parallel. Once a stage completes, then the next stage begins, until the pipeline completes. This is the typical processing loop that GitLab utilizes to complete a CI pipeline.

However, it is possible to create an internal loop of CI jobs that are directed to execute in a specific order that does not loop. This pattern is called a DAG or directed acyclic graph. Directed refers to the ordering of operations, acyclic refers to the fact it only happens once, and graph indicates the ordering of steps.

When leveraged properly, the DAG pattern can dramatically decrease the time it takes a pipeline to complete. This occurs because you are creating processing...

Building code for multiple architectures

GitLab CI enables you to build artifacts for multiple architectures at once. This can easily speed up software builds by two or three times since, with multiple architectures, you typically have to build the software multiple times for each architecture. Here, we’re going to use CI jobs and pipelines to execute a software build for multiple architectures at the same time.

When building software for multiple architectures, there are some special requirements. The first requirement is that you must have GitLab Runners installed and configured on a machine running each architecture. For this section, we’re going to use three platforms as an example: x86_64, arch64, and powerpc. In this case, the expectation would be that you have a machine for each of the three architectures, with a GitLab Runner installed on it. That GitLab Runner also needs to have a tag assigned to it for which architecture it is running on.

The second requirement...

When and how to leverage caching or artifacts

With GitLab, there is often confusion about the usage of caching or artifacts. Many users are curious as to which functionality to leverage and when. We aim to demystify both in this chapter and give you the tools to implement either approach while explaining the benefits and pitfalls of each pattern.

Caching should be thought of as a method to save items that are commonly used in CI jobs or stages. It shouldn’t be thought of as a means to pass items between stages or jobs – that’s what artifacts are for. The difference is important because of the implementation and configuration of each. A cache is not built or designed to be a method to move items between CI jobs. Because of this, in the future, any features or changes that are made to it will be tailored toward that functionality.

Artifacts are the main way to support storing items created by your CI job indefinitely, as well as passing them as a dependency...

Reducing repeated configuration code with anchors and extensions

All GitLab CI pipeline files must be valid YAML. This also means they support various templates and repeatable code patterns that the YAML language supports. Writing multiple CI job definitions for each CI job in a pipeline can be time-consuming and lead to serious maintainability concerns. If you leverage the same variable in multiple places in a pipeline, it’s better to have it defined once, and then referenced multiple times. This way, when you need to change the variable, you do it in one place as opposed to many. Not only is this easier to do, but it’s also safer as with large pipelines, multiple variable replacements can lead to errors.

The three methods GitLab and YAML offer for creating reusable CI pipelines are anchors, the extends (extends:) keyword, and a reference (!reference) tag. In the following sections, we’re going to explain the benefits and usage of each of these three methods...

Improving maintainability by combining multiple pipelines and leveraging parent-child pipelines

Most GitLab users simply utilize a single .gitlab-ci.yml file for their pipelines. This approach is perfectly acceptable, but in many cases, the amount of code inside this file can become very large and difficult to maintain. GitLab has introduced the ability to include multiple gitlab-ci files together as one. In this section, we’re going to cover how to break up a .gitlab-ci.yml file into multiple sections. Later, we’re going to cover how to take a second .gitlab-ci.yml file and execute a second child pipeline, and then discuss the reasons why you may want to do this.

Leveraging includes for maintainability

Create a .gitlab-ci.yml file that looks like this:

"Build Application":
  stage: build
  script:
    - code here
"Build Container":
  stage: build
  script:
    ...

Securing and accelerating jobs with purpose-built containers

GitLab, when set up properly, runs all the CI jobs of a pipeline in a container. This means that the entire build operation happens in a container. Because of this, container stewardship is exceptionally important. If a CI job happens in an insecure container, then that means the entire CI job and pipeline are insecure. If a CI job uses a non-performant container, that CI job and pipeline will take much longer to complete, resulting in a much slower time to show results. In every measurable way, the container used for your CI jobs is the most important part of your pipeline.

Important note

To quickly set or identify which container a specific CI job is using, look for the image: attribute in a CI job. This attribute will define the source of the container image, and the exact container image being used.

A second area to look for this container image is at the top of the CI job log. There will be a message indicating...

Summary

In this chapter, we learned about many tools we can use with GitLab CI to create fast and reusable pipelines. We started with DAGs, which allow us to make pipelines execute faster. Then, we learned how to build code for multiple architectures. With the introduction of mainstream ARM platforms, this is going to gain importance over time. Later, we learned when to leverage caching or artifacts in pipelines for dependency management.

The last two topics we covered are likely the most important. We learned three different methods to build reusable pipeline definitions so that you don’t have to write the same logic in multiple places. Finally, we learned about a concept called purpose-built pipelines, which enables you to build fast, secure, and stable containers to execute your CI workloads in.

In the next chapter, you’re going to learn how to expand the reach of your CI/CD pipelines.