Chapter 5: Jumping into Pipelines

So far, you've seen how to install and use Tekton in your Kubernetes cluster. You've also explored how tasks work and how you can use them to perform various operations inside a pod. Tasks can do some powerful stuff on their own, but they are still somewhat limited. What happens if you want to perform more than one operation? This is where pipelines will come into play. In this chapter, you will learn everything there is to know about pipelines.

First, you will learn a little more about pipelines and where they fit in your CI/CD processes. Once you understand what they are and how they work, you will be introduced to some hands-on examples.

Similar to what was done in the last chapter, you will start by creating a simple Hello-World pipeline to get you accustomed to the general syntax.

Then, we will introduce some more advanced concepts. You will see how to use parameters to make your pipelines more versatile and ensure that...

Technical requirements

You can find all of the examples described in this chapter in the chapter-5 folder of the Git repository: https://github.com/PacktPublishing/Building-CI-CD-systems-using-Tekton.

You can also see the code in action videos at https://bit.ly/3zCNbkk.

Introducing pipelines

Now that you have a better understanding of steps and tasks, it's time to introduce pipelines. Pipelines are a collection of tasks that are designed to produce an output from your inputs.

While a task is in charge of a single operation inside your CI/CD pipeline, a pipeline will use those tasks to perform the desired automation processes on your source code. For example, a pipeline could use a task to clone a code repository and a second task to run unit tests, and finally, have one last task to build an image ready to be deployed on your production servers.

Just like with tasks, you should aim at making your pipelines as reusable as possible. There is a good chance that most of your projects that use a similar tech stack will use the same pipeline, the only difference being the initial code repository and image name that is produced.

Pipelines use YAML files to describe the processes that will occur and the order in which those operations need...

Building your first pipeline

For this first pipeline, you will create a task that will output some text, similar to the hello world task used in the last chapter. You will then use this task to build your first pipeline:

1. First, start with your basic task object in a file called hello-pipeline.yaml:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: first-task 
   spec: 
     steps:

2. By now, you know that you can create a simple step that produces an echo statement using a UBI container. You can keep this step with the minimal information necessary to run:

      - image: registry.access.redhat.com/ubi8/ubi-minimal 
        command: 
          - /bin/bash 
        args: ['-c', 'echo Hello from first task'] 

   Note

   You can add multiple Tekton objects in a YAML file using the --- separator in your file.

3. You can now add the pipeline base definition...

Parameterizing pipelines

Just like with tasks, you can add parameters to your Tekton pipelines. Those parameters would then typically be used as parameters for the tasks inside of them. By using parameters in your pipelines, you will reuse the same pipeline for multiple usages. For example, if you had a CI/CD pipeline to compile a Java application, you could reuse that pipeline across all of your projects, as long as the URL of the Git repository is a parameter.

In this next example, you will rewrite the tasks and pipeline from the last section, but this time keeping reusability in mind:

1. First, create a new file called die-roll-param.yaml and copy over the content from die-roll.yaml.
2. Next, add a parameter for the number of sides for the dice you want to roll. This parameter will be of the string type. You can also add a default value of "6". This parameter takes a string as an argument, so don't forget the quotes around the default value:

     ...

Reusing tasks in the context of a pipeline

In the last chapter, you saw how to make tasks reusable by using parameters. You also saw how you can set the values for those parameters from a pipeline. Using those parameters, you can use a given task in multiple pipelines, and you can even use a given task inside a single pipeline. This is what you will do here:

1. Start with a new file called log-task.yaml. Here, create a new task called logger:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: logger 
   spec: 

2. This task will need a parameter. You can give it the name text. This will be a parameter of the string type:

     params: 
      - name: text 
        type: string 

3. For the steps object, you will have a single step that logs the date and time followed by the text to be logged:

     steps: 
      - name: log 
        image: registry.access.redhat.com/ubi8/ubi-minimal...

Ordering tasks within pipelines

You might have noticed that the task-reuse tasks were coming out in a random order in the last section. Sometimes the say-hello task would run first, while other times it was the log-something task that was completed first. In that specific pipeline, it is tough to predict which task will complete in which order. When the pipeline run is created, the two pods are created at the same time.

In this case, the order in which Tekton executed the tasks didn't matter. Both tasks ran simultaneously in their own pod, and it ended up saving us some time in the total pipeline execution, as opposed to running them one after the other.

Sometimes, though, you will want the tasks to happen in a specific order. This will be especially true when a task produces an output required by a subsequent task.

To demonstrate this, we will create a new pipeline with a single task that we reuse multiple times. The task in question will have three steps. The first...

Using task results in pipelines

In the previous chapter, you've seen how to use results inside tasks to share information across steps. You can also use the same results to share limited information across tasks.

For the last example in this chapter, you will rewrite the dice-roll task to produce a result instead of echoing the result. This randomly generated number will then be picked up by the logger task to be displayed:

1. Start by creating a file named dice-results.yaml in which you will create the base of your new task named dice-roll-result:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: dice-roll-result 
   spec: 

2. Just like you did in the previous version of the die-roll task, add a parameter with a default value of "6":

   params: 
      - name: sides 
        description: Number of sides to the dice 
        default: "6" 
        type: string...

Introducing pipeline runs

Pipeline runs are to pipelines what task runs are to tasks. They are the actual executions of the pipelines.

Using the pipeline used in the last section, let's create a new pipeline run and examine the output:

$ tkn pipeline start results 
? Value for param `sides` of type `string`? (Default is `6`) 6 
PipelineRun started: results-run-sb6lk 
  
In order to track the PipelineRun progress run: 
tkn pipelinerun logs results-run-sb6lk -f -n default 

You can see that when you run the tkn pipeline start command, it generates a pipeline run with a random name. In this case, the name is results-run-sb6lk. To see the output of the run, you will use the tkn CLI tool to visualize the logs of this specific pipeline run.

Because pipeline runs are Kubernetes objects, you can manipulate them with kubectl the same way you would any other Kubernetes primitive. For example, you could use kubectl get to list all the pipeline runs that exist inside...

Getting your hands dirty

Once again, it is time for you to test out those newly learned skills. You can find a solution to these problems in the Assessments chapter of this book. These challenges will use the concepts that you have seen in this chapter.

Back to the basics

For your first challenge, start by building a pipeline that will output a hello message. To do so, use the task called logger that you created in this chapter. The pipeline will take a parameter to indicate who to say hello to, which should default to World. Run this pipeline with different parameter values using the CLI. Run it with the default values. Do one last run with the parameter value specified in the CLI command directly.

Tips

- This pipeline has a single task.

- The parameter is passed from the pipeline to the task.

- Reuse the previously created task logger.

Counting files in a repo

For this next challenge, build a pipeline that will clone a Git repository and then output...

Summary

You now know everything there is to know about pipelines and how you can use them to run various tasks. You've seen how you can pass parameters across the tasks. You have also seen how tasks can be reused with different parameters to produce different outputs.

You can run those tasks either in parallel or in sequence, and you've seen how to use a mix of both to make your pipelines more efficient.

After you learned how to order your tasks, you saw how tasks can output some results that can then be used by a task that comes later in the pipeline execution.

You have also learned how to analyze pipeline runs and visualize them to help you find what is wrong with your pipeline executions when they fail. In the next chapter, you will learn more advanced techniques to debug your tasks and pipelines.