Chapter 7: Sharing Data with Workspaces

In the last few chapters, you've seen some of Tekton's basic elements. With Tekton installed on your Kubernetes cluster, you can create tasks, which are small operations that you can perform as part of a pipeline. You've also seen how you can chain those tasks as part of a more extensive pipeline. So far, the examples that you've built have been relatively simple. The main limitation, for now, had to do with sharing data across the tasks. You've seen how to use results to transmit small bits of data, but you will need to share entire code bases if you want to automate operations on them. This is a job for workspaces, and you will learn how to use them in this chapter.

First, you will see how to use workspaces in the context of a single task. Using a workspace like this will let you share a volume across all the steps that compose your task. In this volume, you will be able to put larger pieces of data than you were...

Technical requirements

You can find all of the examples described in this chapter in the chapter-7 folder of the Git repository: https://github.com/PacktPublishing/Building-CI-CD-systems-using-Tekton.

You can also see the code in action videos at: https://bit.ly/3kRAKNA

Introducing workspaces

Workspaces are shared volumes used to transfer data between the various steps of a task. When authoring a new task to be used by your team, this has the benefit of making your tasks more flexible and reusable in different contexts. Using workspaces, your tasks' users will pick the type of volume they want to use for this volume.

You can also use workspaces to share data across tasks in a Tekton pipeline. This will come in handy when you want to share multiple files across the various steps of a pipeline. So far, you've used results to exchange small bits of information limited to 4,096 bytes, but eventually, you will want to share code bases and images, and this will be done with a workspace.

Types of volume sources

Workspaces need some sort of volume to mount the files to be shared. These types of volumes are called VolumeSources. The volume source you will use with your workspace will depend on the goals you want to achieve. The examples that will follow in this chapter will mainly target emptyDir and persistent volume claims. Still, four different sources are available to you when you author your tasks and pipelines.

emptyDir

emptyDir is, as the name suggests, an empty directory that is attached to a task run. This temporary folder should only be used in the context of a task and is not suitable to share information across tasks in a pipeline. It is, however, a straightforward mechanism that you can use to share data across steps in a task. This approach should be prioritized over using the home folder, as demonstrated in Chapter 4, Stepping into Tasks.

ConfigMap

The config map source lets your users use a Kubernetes-native ConfigMap object as a workspace...

Using your first workspace

For this first workspace example, you will clone a Git repository and share this repository's content to a second step to list this folder's content:

1. First, start with a new file called clone-ls.yaml. This file will contain a new task named clone-and-list:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: clone-and-list 
   spec: 

2. To make this task as generic as possible, it will take a parameter for the repository's URL to clone. You can specify any Git repository as the default value:

    params:  
      - name: repo 
        type: string 
        description: Git repository to be cloned 
        default: https://github.com/joellord/handson-tekton 

3. Next, you will need to define workspaces for this task. In this task, you will use a single workspace with the name source:

    workspaces: 
      - name: source 

...

Using workspaces with task runs

The previous task worked well when initiated with the tkn CLI tool, where you had to fill in all the information. Sometimes, you might want to have all of that information as part of a YAML file.

Unlike parameters with default values that you could use, you will need to specify the name and volume source for each workspace you have in your tasks. One way to store this information as part of your YAML files is to write a task run directly. This file will be used as a template to create your task runs. Let's see how to do this:

1. Start with a new file called clone-and-list-tr.yaml. In this file, create a new object of kind TaskRun:

   apiVersion: tekton.dev/v1beta1 
   kind: TaskRun 

2. In the metadata section of this object description, you could potentially specify a name for the task run. A unique name would mean that you would need to change the name each time you want to create a new task run, though. Instead, you will use the generateName...

Adding a workspace to a pipeline

So far, you've managed to share a workspace across the steps of a task. As you build some real CI/CD pipelines for your applications, you will most likely need to share some data across your pipeline tasks. A typical pipeline would start with cloning a repository, and the other tasks would perform some operations on that code base. You can do this by using a workspace at the pipeline level.

For this example, you will split the task you just created into two separate tasks. This first task clones the source code, and the entire code base is shared with the list task:

1. In a file called split-tasks.yaml, copy over the task from the clone-ls.yaml file and change the name to clone. You can also remove the last step as it will now be part of a new task:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: clone 
   spec: 
     params:  
       - name: repo 
        &...

Persisting data within a pipeline

To persist the data across your tasks, you will need to look into persistent volumes and persistent volume claims. Those are Kubernetes native objects that can be used to access some disk space on a cluster. The exact way to create a persistent volume claim in your cluster might differ from the following example. Please refer to the documentation of your cloud provider for more information. Most providers will provide you with a dynamic assignment and won't need a persistent volume.

Note

If you want to learn more about how PersistentVolumes and PersistentVolumeClaims work and manage them, look up the Kubernetes official documentation on the subject at https://kubernetes.io/docs/concepts/storage/.

For this example, I am using minikube, which requires a persistent volume that can then be claimed to be used by objects in the cluster:

1. In a new file called persistent-volume.yaml, create a new object of kind PersistentVolume. This...

Cleaning up with finally

On that last pipeline run, a fatal error occurred. It tried to clone the repository, but Git requires an empty directory to clone the files into. The folder already had files from the previous clone operation.

To avoid those errors and ensure that you always start from a clean slate, you will need to clean up your persisting workspaces when the pipeline run has terminated. To do so, you will need to add a cleanup task. You will also need to ensure this task runs at the end of the pipeline, even if an error terminated the pipeline before it completed its run. This is where the finally tasks will come into play.

In this section, you will add a cleanup task to your pipeline that will remove all the files from the workspace:

1. Create a new file called cleanup.yaml. In here, create a new task named cleanup:

   apiVersion: tekton.dev/v1beta1 
   kind: Task 
   metadata: 
     name: cleanup 
   spec: 

2. Just like the other tasks you've built in this...

Using workspaces in pipeline runs

As part of your pipeline definition, you cannot add a workspace the same way you could add default values to your parameters. If you want Tekton to start your pipeline automatically with the appropriate persistent volume claim, you will need to create a pipeline run:

1. Start with a new file called pipelinerun.yaml. In there, create an object of kind PipelineRun. Like what you did with task runs, this object will have a generateName field instead of an actual name:

   apiVersion: tekton.dev/v1beta1 
   kind: PipelineRun 
   metadata: 
     generateName: clone-and-ls-pr- 
   spec: 

2. This pipeline run will use the pipeline that you just built called clone-and-list:

     pipelineRef: 
       name: clone-and-list 

3. Finally, add a workspace named codebase. This workspace uses a persistent volume claim, and the claimName is tekton-pvc:

    workspaces: 
      - name: codebase 
        persistentVolumeClaim...

Using volume claim templates

Instead of specifying a persistent volume claim directly, you can also ask Tekton to create a temporary one for you. This can be useful when you don't need to persist data outside of your pipelines. As an additional benefit, using a volume claim template enables you to run multiple pipeline runs concurrently. To use a volumeClaimTemplate, you will need to write a new pipeline run. This time, you will add the pipelineSpec, which is a way to define your entire pipeline without relying on a separate pipeline object:

1. Start with a new file called pvc-template.yaml. In there, create a new PipelineRun object:

   apiVersion: tekton.dev/v1beta1 
   kind: PipelineRun 
   metadata: 
     generateName: clone-and-ls-pr-  

2. In the spec field of the pipeline run, add a pipelineSpec field containing a workspaces list with one workspace called codebase:

   spec:
     pipelineSpec: 
       workspaces: 
       ...

Getting your hands dirty

Data sharing across tasks and pipelines should have no secrets for you now. You are ready to test your knowledge with the following exercises. In order to complete those challenges, you will need to use the concepts introduced here, along with notions from the previous chapters. The solutions to these exercises can be found in the Assessments section at the end of the book.

Write and read

Create a task that uses a workspace to share information across its two steps. The first step will write a message, specified in a parameter, to a file in the workspace. The second step will output the content of the file in the logs. Try running the task using the -w parameter in the tkn CLI tool.

Tips

- Use the variable substitutions to get the path of the workspace and append a file name.

- Use the cat command to output the content of a file.

- You can use an emptyDir volume when the data is shared within a task.

Pick a card

Using the Deck...

Summary

You now know how to persist data and use shared volumes across various tasks in a Tekton pipeline. Now that you can share information such as code bases or images, it will be easier to make examples closer to your day-to-day life as a software developer and CI/CD author.

You've also seen how you can build task runs and pipeline runs directly and pass them arguments such as the parameter values or the workspace definition.

If you've done the exercises, more specifically the last one, you had to use some if statements in your Bash scripts to validate the user role. In Tekton, there is a way to perform some tasks based on specific conditions instead of relying on Bash scripting. These are called when expressions and will be introduced in the next chapter.