Persisting data within a pipeline

To persist the data across your tasks, you will need to look into persistent volumes and persistent volume claims. Those are Kubernetes native objects that can be used to access some disk space on a cluster. The exact way to create a persistent volume claim in your cluster might differ from the following example. Please refer to the documentation of your cloud provider for more information. Most providers will provide you with a dynamic assignment and won't need a persistent volume.

Note

If you want to learn more about how PersistentVolumes and PersistentVolumeClaims work and manage them, look up the Kubernetes official documentation on the subject at https://kubernetes.io/docs/concepts/storage/.

For this example, I am using minikube, which requires a persistent volume that can then be claimed to be used by objects in the cluster:

1. In a new file called persistent-volume.yaml, create a new object of kind PersistentVolume. This...