Most transactions and interactions today are digital or rely on digital services, so the ability to deliver software quickly, reliably, and securely is a critical competitive advantage. Continuous delivery (CD) is a software engineering practice that enables teams to achieve this goal by automating the entire software release process, from committing code to deploying to production. CD, properly done, empowers organizations to reduce their time to market (TTM), improve software quality, and release new features and bug fixes more frequently.

The goal of this book is to share how to implement secure CD using Google Cloud services such as Cloud Code, Cloud Workstations, Cloud Build, Artifact Registry, Cloud Deploy, and others.

In this book, we describe each service and show how you can use them together to automate and secure your software delivery pipeline. But first, in this chapter, we introduce CD principles and some of the practices and technical capabilities that drive the ability to implement them.

This chapter also provides an overview of security threats that a software supply chain can face and some practices that can mitigate those threats.

This chapter includes the following sections:

• Introduction to CD
• Understanding continuous integration
• Understanding continuous testing
• Understanding deployment automation
• Securing your software delivery pipeline

We start this book by looking at what CD is.

CD is a set of practices and principles that aim to streamline and automate software delivery from commit to production. Teams and organizations practice with the goal of deploying software to production on demand at any time without impacting service availability. The main objective of CD is that software should always be in a deployable state and software release should be a fast, repeatable process.

This section is a quick overview of CD, what it consists of, and how it can help your organization achieve better and faster software delivery.

CD practices

Some of the practices that underpin CD are set out here:

• CI: CI means that code changes are integrated into the source code repository quickly and regularly. Automated builds and tests are triggered with each code commit, providing rapid feedback to developers. Consistent, reliable builds are at the foundation of a trustworthy CD process.
• Continuous testing: This includes unit tests, integration tests, and end-to-end (E2E) tests. Test suites are executed automatically during the pipeline to catch regressions early and increase your confidence in the quality of the system.
• Small, frequent releases: Instead of large, infrequent releases, CD tries to break down features and changes into small, manageable chunks that can be released more frequently. This reduces the risk of introducing regressions and makes it easier to identify and fix problems.
• Trunk-based development: A software development methodology in which developers divide their work into small batches and merge each batch into the trunk at least once a day. This approach is in contrast to more complex branching strategies such as feature branching or Gitflow, in which developers create separate branches for different features or bug fixes.
• Deployment pipeline: The entire release process, from building the code to deploying it to production, is automated using a pipeline. The automated pipeline can run such tasks as the following:
  • Compiling code
  • Executing unit tests
  • Building software artifacts as container images
  • Progressively deploying the app on different pre-production environments where different kinds of automated or manual tests can be performed
  • Finally releasing the application to production

The following diagram represents an example pipeline, starting when there is a new commit in the source code repository. The tasks represented in the diagram and the ones listed previously are examples. The exact pipeline sequence changes depending on factors such as the application itself, the programming language, and the framework, as well as specific organizational contexts:

Figure 1.1 – A high-level look at a software delivery pipeline

That process, with those practices, can help you achieve improved software delivery performance, as described in the next section.

The impact of CD on software delivery performance

The DevOps Research and Assessment (DORA) research program has identified CD as one of the main capabilities driving software delivery performance, as measured by DORA’s four key metrics.

The following list shows those four metrics and how they can help you improve your software delivery performance:

• Deployment Frequency: How often code changes are deployed to production.

  CD encourages more frequent deployments because it automates the deployment process and ensures that code changes are always production ready. Teams practicing CD can deploy changes to production on demand, often multiple times a day. Compare this to traditional approaches that might have longer release cycles.

• Lead Time for Changes: The time it takes to go from code commit to a production-ready release.

  CD streamlines the software development process, enabling faster development cycles and reducing delays. CD automates steps in the delivery pipeline, such as building, testing, and deploying, which reduces manual intervention and wait times.

  With CD, code changes are continuously integrated, tested, and delivered, shortening the time from development to production. Frequent, small releases accelerate TTM.

• Change Failure Rate: The percentage of changes or deployments that fail or require rollback.

  CD emphasizes continuous testing, including unit tests, integration tests, and acceptance tests. This reduces the chance that defects and errors make their way into production.

  Automation, in general, makes the release process repeatable and less error-prone because almost nothing is left to interpretation. Frequent, smaller releases make it easier to identify and fix issues early in the development process, reducing the likelihood of catastrophic failures in production.

• Time to Restore Services: The average time it takes to restore services after a production failure.

  CD practices typically include automated monitoring and alerting in production environments. This helps teams detect issues quickly. When issues do occur, CD enables rapid rollback or forward fixes. Automated deployments make it easier to apply fixes and quickly get the system back to a working state.

See https://dora.dev/ for more info on DORA and the four key metrics.

In the rest of this chapter, we describe the most important underlying CD practices in more detail, starting with CI.

Continuous integration (CI) is a set of software development practices for frequently integrating code changes. Each time a developer integrates changes (usually at least daily), those changes are automatically verified and tested. This means that there is always an up-to-date build ready to deploy.

The practice of CI was created to solve integration and compatibility problems caused by the old approach of developers working in isolation for an extended period on their version of code and then attempting to merge their changes into a shared code base.

CI follows the counterintuitive principle that if something is painful, you can make it less painful by doing it more often. This is based on the understanding that the integration effort is exponentially proportional to the amount of time between integrations (as found on https://martinfowler.com/articles/originalContinuousIntegration.html#TheMoreOftenTheBetter).

The next section describes how to bring these principles to your software development process.

How to implement CI

Here is a list of fundamental practices that a team should consider following to implement CI:

• Trunk-based development: As described previously, developers divide their own work into small batches and merge that work into the main code branch (trunk) at least once a day.
• Automated builds: For every change committed to the repository, the application is automatically compiled, built, and tested without manual intervention. This helps ensure that the software is always in a state that can be deployed to production. Artifacts produced by the CI build should be used by all subsequent deployment tasks. An automatic build is run at least once a day.
• Continuous testing: Automated tests run every time a change is committed, including unit tests, integration tests, and other types of automated tests. This helps ensure that new code doesn’t break existing functionality. Immediate feedback from these tests allows developers to fix issues quickly before those issues cause bigger problems. Tests should run successfully at least once a day, and if they fail, the team prioritizes fixing the problem.
• Collaboration: This is probably more a consequence of the previously listed elements than a principle to implement. With all team members committing to the shared repository regularly, there’s a continual exchange of ideas and code. This openness not only improves the quality of the software but also enhances team dynamics, as everyone can see what others are working on. It also encourages developers, testers, and operations teams to work more closely together. This collaboration breaks down silos and promotes a more cohesive approach to software development, in which everyone is aware of the goals and challenges of the project.

Now that we’ve covered the main elements of CI, let’s explore continuous testing.

Continuous testing automates as much of the testing as possible, running the tests frequently (before and after each build) as part of a CD pipeline.

This approach has the following benefits:

• Developers and the rest of the team get quick feedback on how the software is functioning and can immediately fix any issues.
• Tests’ reliability is better than that of manual tests, which are repetitive and error-prone.
• Software can be released more frequently because the feedback loop is shorter.

Using continuous, automated testing doesn’t mean you won’t also run some manual tests, such as exploratory or usability testing, throughout the delivery process.

Now, let’s look at the types of testing that are typically included in continuous testing.

Test types

The following are some test types typically used in continuous testing:

• Unit tests: These tests are building blocks, focusing on individual code units (functions, classes). Unit tests run quickly and provide fine-grained feedback.
• Integration tests: These tests verify how different parts of a system work together. Integration tests are crucial for identifying issues in the interactions between integrated components or systems. These are more likely to be run within your CI server where multiple code packages come together.
• Acceptance tests: These tests simulate real-world usage of a running application or service, verifying that essential functionality aligns with user expectations and business requirements. These may happen from your CI environment or other outside services that test applications running in a staging environment.

And now, let’s see how to make continuous testing happen.

Implementing continuous testing

The following is a list of fundamental principles and practices for teams to implement continuous testing:

• Test automation: Automating unit tests, integration tests, and regression tests is essential for continuous testing. Automation allows the team to quickly identify and address issues, making the testing process more efficient.
• Developer involvement in testing: Developers should be the people primarily responsible for creating and maintaining automated test suites.

  This approach ensures that tests are always updated based on code changes and that developers write code that is relatively easy to test. The natural consequence and best realization of this approach is the test-driven development (TDD) practice, which is described next.

• TDD: A software development methodology in which tests are written before code. Initially, the developer writes an automated test for a new function or feature and then writes the actual code. This test defines how the new functionality should behave.

  The developer then writes the minimal amount of code necessary to make the test pass. After the test passes, the developer then refactors the code, which involves cleaning up and optimizing the code without changing its functionality.

• Unit and acceptance tests’ proportion: One of the main goals of continuous testing is to detect and fix issues as soon as possible. Unit tests are typically faster than acceptance tests and are executed in the early phases, so the more issues are detected in unit tests, the sooner defects are detected and remediated.

  As much as possible, unit tests should do the work of detecting issues. When an error is found in acceptance tests or other manual tests, create an automated unit test for that behavior to ensure that this error is detected sooner if it occurs again.

Now that we’ve covered the main elements of continuous testing, we’ll explore automated deployment in the next section.

Deployment automation allows software to be easily and automatically deployed to pre-production and production environments, at the push of a button. This is crucial for minimizing risks during production rollouts and results in rapid feedback on software quality. By enabling teams to perform extensive testing immediately after updates, deployment automation ensures prompt assessment and enhances overall software quality. Implementing deployment automation doesn’t necessarily mean that you are doing continuous deployment. Continuous deployment is when you try to deploy every code change to production as soon as possible, and this can or cannot be applicable to your environment. You can do CD and automate your deployments but still have gated releases, feature flags, or even manual switches to formally light up the release.

An automated deployment process has the following inputs:

• Software artifacts (created by CI) that are ready for deployment in any environment, including production
• Configuration details unique to each environment
• Scripts for setting up the environment, deploying packages, and conducting deployment tests

Now, let’s look at how to automate your deployments.

Implementing deployment automation

Here is a list of fundamental principles and practices that a team should follow to implement deployment automation:

• Implement a uniform deployment process across all environments, including production. This ensures the deployment method is thoroughly tested before it’s used in production.
• Enable automated deployment of any artifact version to any environment by anyone with the right permissions. A process is not fully automated if it requires that someone file a ticket or manually prepare an environment.
• Use identical packages across all environments and maintain environment-specific configurations separately. This ensures packages deployed in production are the same as those tested.
• Ensure each environment’s state can be re-created using version control. This makes deployment repeatable and enables deterministic production restoration in disaster recovery (DR) scenarios.

That covers deployment automation. In the next section, we introduce software delivery security.

Software production and delivery can be viewed as a supply chain in which the same developer can be both a software producer (by developing software) and a consumer (by using existing software as a dependency for the app being developed).

This section is a high-level overview of some of the security threats against a software delivery pipeline, as well as some best practices that help protect against those threats. In later chapters, we describe how to implement some of the best practices, using Google Cloud-managed services.

First, let’s look at the security of your source code.

Source code management threats and remediations

Threats to source code management systems affect the security of the source code. These threats can come from the code itself (for example, submitting source code that is unintentionally or intentionally vulnerable) or can be threats to the source control platform, which can be compromised in different ways. The next section lists some practices to help ensure your source code is safe.

Source code management security best practices

The following is a list of security best practices to protect the integrity of your source code:

• Repository configuration: Configure repositories using a secure, automated method that offers preconfigured templates based on the application’s security level.

  Enforce centralized identity management with multi-factor authentication (MFA) for users, ensuring updated access privileges. Limit repository ownership to a few trusted employees, integrate identity management systems, and require dual approval for merges to enhance security.

• Code reviews: Code reviews are one of the most important practices to ensure software quality and security and should always be part of a secure software supply chain.

  Reviewers should be assigned to a repository application or a specific commit based on expertise in the programming language and relevant security risks. Perform code reviews on feature-branch pull requests as soon as developers are ready.

  In addition to human code reviews, implement static code analysis tools.

• Merge approvals: Establish a select group of trusted individuals to authorize merges into production branches. For applications with stricter security requirements, implement a process that requires more than one approver.

Now, let’s look at security for your build process.

Build threats and remediations

The following are some examples of security threats to the artifact build process and tools:

• Builds that use modified source code (not from a trusted source control system)
• Compromise of the build system
• Artifacts built outside of the official build system and process
• Compromise of the artifact storage system

The following section lists some practices to help ensure the security of your build process.

Build security best practices

The Supply-chain Levels for Software Artifacts (SLSA) specification defines a set of build-security best practices. These are established by multiple software-industry organizations under the Open Source Security Foundation (OpenSSF) (https://openssf.org/).

SLSA (http://slsa.dev) is a specification meant to help you describe and improve your software supply chain security. It’s structured as a series of tracks, each one covering an aspect of supply chain security, and levels, with each level offering progressively stronger security guidelines and requirements for each track.

At the time of this writing, the SLSA specification is at version 1.0, the first stable release of SLSA. SLSA v1.0 consists of only one track: Build. Future versions of SLSA will add tracks that cover other parts of the software supply chain.

Each level includes a set of requirements for the build process: lower-level requirements are easier to implement but provide lower security guarantees. Higher-level requirements are harder and usually more expensive to adopt but demonstrate that tougher security guidelines have been adopted for software build practices.

The level-based structure also allows you to incrementally adopt the guidelines, progressively improving the security posture of your software supply chain.

The primary goal of the SLSA Build track is to attest and verify that an artifact was built as expected. This is done mainly by generating build provenance, which software consumers can verify. The SLSA v1.0 Build track includes three levels: L1 to L3. There is also an L0 level, which refers to software that doesn’t meet any SLSA requirements. The following table summarizes the build levels:

Table 1.1 – SLSA v1 Build tracks

These security levels, and descriptions thereof, are from the SLSA specification, version 1.0, courtesy of the SLSA Working Group. You can find more details on the requirements for each level at http://slsa.dev.

In the next section, we’ll talk about threats in your code’s dependencies and some practices for remediating those threats.

Dependency threats and remediations

Dependency threats are vulnerabilities or intentionally malicious code in any software that your application depends on.

Dependency management best practices

The following is a list of security best practices for managing dependencies:

• Scan dependencies for vulnerabilities: Integrate a scanning tool into the development and build workflow that identifies vulnerabilities, using databases such as the National Institute of Standards of Technology National Vulnerability Database (NIST NVD).
• Store dependencies in a private registry: A private registry offers the convenience of a public repository, with enhanced control over dependencies. It can provide capabilities such as access control, vulnerability scanning, and repository management.
• Remove unused dependencies: Regularly cleaning up unneeded dependencies keeps your application lean and nimble, improving its performance and reducing its attack surface. This proactive approach helps minimize potential security risks associated with outdated or irrelevant dependencies.
• Use open source tools to track dependency insights: Various tools can help you to get information on dependencies. Here are two examples:
  • Open Source Insights (https://deps.dev/): A website that provides information about software packages vulnerabilities and licenses.
  • Open Source Vulnerability (OSV) Database (https://osv.dev/): A searchable vulnerability database.

This section described how to secure your software delivery pipeline, including a look at how your code and its dependencies can face vulnerabilities.

In this chapter, you were introduced to CD principles and some of the practices and technical capabilities that underpin them, such as CI, continuous testing, deployment automation, and trunk-based development. You also learned how CD and these practices positively impact software delivery performance and achieve organization goals, as measured by DORA research.

In the second part of the chapter, we described some potential security threats to a software supply chain and some strategies to protect against them. We also provided an overview of the SLSA Build track specification.

This information should prepare you for the rest of the book, in which we show how to implement CD and software supply chain security best practices using Google Cloud-managed services.

The next chapter is a look at Skaffold, an open source tool for continuous development and CD that is used by several of the Google Cloud services described in this book.