Chapter 6. Background Processing in Kubernetes

Kubernetes includes support for one-off (also known as batch) computation work, as well as supporting common use cases for asynchronous background work. In this chapter, we look at the Kubernetes concept of job, and its neighbor, CronJob. We also look at how Kubernetes handles and supports persistence, and some of the options that are available within Kubernetes. We then look at how Kubernetes can support asynchronous background tasks and the ways those can be represented, operated, and tracked by Kubernetes. We also go over how to set up worker codes operating from a message queue.

Topics covered in this chapter include:

• Job
• CronJob
• A worker queue example with Python and Celery
• Persistence with Kubernetes
• Stateful Sets
• Custom Resource Definitions (CRDs)

Job

Most of what we have covered so far has been focused on continuous, long-running processes. Kubernetes also has support for shorter, discrete runs of software. A job in Kubernetes is focused on a discrete run that is expected to end within some reasonably-known timeframe, and report a success or failure. Jobs use and build upon the same construct as the long-running software, so they use the pod specification at their heart, and add the concept of tracking the number of successful completions.

The simplest use case is to run a single pod to completion, letting Kubernetes handle any failures due to a node failure or reboot. The two optional settings you can use with jobs are parallelism and completion. Without specifying parallelism, the default is 1 and only one job will be scheduled at a time. You can specify both values as integers to run a number of jobs in parallel to achieve multiple completions, and you can leave completions unset if the job is working from a work queue of some...

CronJob

CronJobs are an extension that build on jobs to allow you to specify a recurring schedule for when they run. The name pulls from a common Linux utility for scheduling recurring scripts called cron. CronJobs were alpha as of Kubernetes version 1.7, and moved to beta in version 1.8, and remain in beta as of version 1.9. Remember that Kubernetes specifications may change, but tend to be fairly solid and have expected utility with beta, so the v1 release of CronJobs may be different, but you can likely expect it to be pretty close to what's available as of this writing.

The specification is highly related to a job, with the primary difference being the kind is CronJob and there is a required field schedule that takes a string representing the timing for running this job.

The format for this string is five numbers, and wildcards can be used. The fields represent:

• Minute (0–59)
• Hour (0–23)
• Day of Month (1–31)
• Month (1–12)
• Day of Week (0–6)

A * or? a character can be used in any of these fields...

A worker queue example with Python and Celery

Where the CronJob is well positioned to run repeated tasks at a specific schedule, another common need is to process a series of work items more or less constantly. A job is well oriented to running a single task until it is complete, but if the volume of things you need to process is large enough, it may be far more effective to maintain a constant process to work on those items.

A common pattern to accommodate this kind of work uses a message queue, as shown here:

With a message queue, you can have an API frontend that creates the work to be run asynchronously, move that into a queue, and then have a number of worker processes pull from the queue to do the relevant work. Amazon has a web-based service supporting exactly this pattern of processing called Simple Queue Service (SQS). A huge benefit of this pattern is decoupling the workers from the request, so you can scale each of those pieces independently, as required.

You can do exactly the same...

Persistence with Kubernetes

So far, all our examples, and even code, have been essentially stateless. In the last chapter, we introduced a container using Redis, but didn't specify anything special for it. By default, Kubernetes will assume any resources associated with a pod are ephemeral, and if the node fails, or a deployment is deleted, all the associated resources can and will be deleted with it.

That said, almost all the work we do requires storing and maintaining state somewhere—a database, an object store, or even a persistent, in-memory queue. Kubernetes includes support for persistence, and as of this writing, it's still changing and evolving fairly rapidly.

Volumes

The earliest support in Kubernetes was for volumes, which can be defined by the cluster administrator, and we've already seen some variations of this construct with the configuration being exposed into a container using the Downward API back in Chapter 4, Declarative Infrastructure.

Another kind of volume that can be easily...

Stateful Sets

Following dynamic provisioning, as you think about persistence systems – whether they are classic databases, key-value data stores, memory caches, or document-based datastores – it is common to want to have some manner of redundancy and failover. ReplicaSets and deployments go a fairly significant way to supporting some of that capability, especially with persistent volumes, but it would be greatly beneficial to these systems to have them more fully integrated with Kubernetes, so that we can leverage Kubernetes to handle the life cycle and coordination of these systems. A starting point for this effort is Stateful Sets, which act similarly to a deployment and ReplicaSet in that they manage a group of pods.

Stateful Sets differ from the other systems as they also support each pod having a stable, unique identity and specific ordered scaling, both up and down. Stateful Sets are relatively new in Kubernetes, first appearing in Kubernetes 1.5, and moving into beta in version 1.9...

Custom Resource Definition

Stateful Sets don't automatically match all the persistent stores that are available, and some of them have even more complex logic requirements for managing the life cycle of the application. As Kubernetes looked at how to support extending its controllers to support more complex logic, the project started with the idea of Operators, external code that could be included in the Kubernetes project, and has evolved as of Kubernetes 1.8 to make this more explicit with CustomResourceDefinitions. A custom resource extends the Kubernetes API, and allows for custom API objects to be created and matched with a custom controller that you can also load into Kubernetes to handle the life cycle of those objects.

Note

Custom Resource Definitions go beyond the scope of what we will cover in this book, although you should be aware that they exist. You can get more details about Custom Resource Definitions and how to extend Kubernetes at the project's documentation site: https:...

Summary

In this chapter, we reviewed jobs and CronJobs, which Kubernetes provides to support batch and scheduled batch processing, respectively. We also looked through a Python example of how to set up a Celery worker queue with RabbitMQ and configure the two deployments to work together. We then looked at how Kubernetes can provide persistence with volumes, PersistentVolume, and its concept of PersistentVolumeClaims for automatically creating volumes for deployments as needed. Kubernetes also supports Stateful Sets for a variation of deployment that requires stable identity and persistent volumes, and we looked at a simple Node.js example converting our previous example of a deployment into a Stateful Set. We finished the chapter with a look at Custom Resource Definitions, used to extend Kubernetes.

In the next chapter, we start to look at how to leverage Kubernetes to get information about all these structures. We review how to capture and view metrics, leveraging Kubernetes and additional...