Using Kubernetes Resources in Practice

In this chapter, we will design a fictional massive-scale platform that will challenge Kubernetes’ capabilities and scalability. The Hue platform is all about creating an omniscient and omnipotent digital assistant. Hue is a digital extension of you. Hue will help you do anything, find anything, and, in many cases, will do a lot on your behalf. It will obviously need to store a lot of information, integrate with many external services, respond to notifications and events, and be smart about interacting with you.

We will take the opportunity in this chapter to get to know kubectl and related tools a little better and explore in detail familiar resources we’ve seen before, such as pods, as well as new resources, such as jobs. We will explore advanced scheduling and resource management.

This chapter will cover the following topics:

• Designing the Hue platform
• Using Kubernetes to build the Hue platform
...

Designing the Hue platform

In this section, we will set the stage and define the scope of the amazing Hue platform. Hue is not Big Brother; Hue is Little Brother! Hue will do whatever you allow it to do. Hue will be able to do a lot, which might concern some people, but you get to pick how much or how little Hue can help you with. Get ready for a wild ride!

Defining the scope of Hue

Hue will manage your digital persona. It will know you better than you know yourself. Here is a list of some of the services Hue can manage and help you with:

• Search and content aggregation
• Medical – electronic heath records, DNA sequencing
• Smart homes
• Finance – banking, savings, retirement, investing
• Office
• Social
• Travel
• Wellbeing
• Family

Let’s look at some of the capabilities of the Hue platform, such as smart reminders and notifications, security, identity, and privacy.

Smart reminders and notifications...

Using Kubernetes to build the Hue platform

In this section, we will look at various Kubernetes resources and how they can help us build Hue. First, we’ll get to know the versatile kubectl a little better, then we will look at how to run long-running processes in Kubernetes, exposing services internally and externally, using namespaces to limit access, launching ad hoc jobs, and mixing in non-cluster components. Obviously, Hue is a huge project, so we will demonstrate the ideas on a local cluster and not actually build a real Hue Kubernetes cluster. Consider it primarily a thought experiment. If you wish to explore building a real microservice-based distributed system on Kubernetes, check out Hands-On Microservices with Kubernetes: https://www.packtpub.com/product/hands-on-microservices-with-kubernetes/9781789805468.

Using kubectl effectively

kubectl is your Swiss Army knife. It can do pretty much anything around a cluster. Under the hood, kubectl connects to your cluster...

Separating internal and external services

Internal services are services that are accessed directly only by other services or jobs in the cluster (or administrators that log in and run ad hoc tools). There are also workloads that are not accessed at all. These workloads may watch for some events and perform their function without exposing any API.

But some services need to be exposed to users or external programs. Let’s look at a fake Hue service that manages a list of reminders for a user. It doesn’t really do much – just returns a fixed list of reminders – but we’ll use it to illustrate how to expose services. I already pushed a hue-reminders image to Docker Hub:

docker push g1g1/hue-reminders:3.0

Deploying an internal service

Here is the deployment, which is very similar to the hue-learner deployment, except that I dropped the annotations, env, and resources sections, kept just one or two labels to save space, and added a ports...

Advanced scheduling

One of the strongest suits of Kubernetes is its powerful yet flexible scheduler. The job of the scheduler, put simply, is to choose nodes to run newly created pods. In theory, the scheduler could even move existing pods around between nodes, but in practice, it doesn’t do that at the moment and instead leaves this functionality for other components.

By default, the scheduler follows several guiding principles, including:

• Split pods from the same replica set or stateful set across nodes
• Schedule pods to nodes that have enough resources to satisfy the pod requests
• Balance out the overall resource utilization of nodes

This is pretty good default behavior, but sometimes you may want better control over specific pod placement. Kubernetes 1.6 introduced several advanced scheduling options that give you fine-grained control over which pods are scheduled or not scheduled on which nodes as well as which pods are to be scheduled...

Join our book community on Discord

https://packt.link/K8s

In Chapter 3, High Availability and Reliability, we looked at reliable and highly available Kubernetes clusters, the basic concepts, the best practices, and the many design trade-offs regarding scalability, performance and cost.

In this chapter, we will explore the important topic of security. Kubernetes clusters are complicated systems composed of multiple layers of interacting components. Isolation and compartmentalization of different layers is very important when running critical applications. To secure the system and ensure proper access to resources, capabilities, and data, we must first understand the unique challenges facing Kubernetes as a general-purpose orchestration platform that runs unknown workloads. Then we can take advantage of various securities, isolation, and access control mechanisms to make sure the cluster, the applications running on it, and the data are all safe. We will discuss various best practices...

Understanding Kubernetes security challenges

Kubernetes is a very flexible system that manages very low-level resources in a generic way. Kubernetes itself can be deployed on many operating systems and hardware or virtual-machine solutions, on-premises, or in the cloud. Kubernetes runs workloads implemented by runtimes it interacts with through a well-defined runtime interface, but without understanding how they are implemented. Kubernetes manipulates critical resources such as networking, DNS, and resource allocation on behalf of or in service of applications it knows nothing about. This means that Kubernetes is faced with the difficult task of providing good security mechanisms and capabilities in a way that application developers and cluster administrators can utilize, while protecting itself, the developers, and the administrators from common mistakes.

In this section, we will discuss security challenges in several layers or components of a Kubernetes cluster: nodes, network, images...

Hardening Kubernetes

The previous section cataloged and listed the variety of security challenges facing developers and administrators deploying and maintaining Kubernetes clusters. In this section, we will hone in on the design aspects, mechanisms, and features offered by Kubernetes to address some of the challenges. You can get to a pretty good state of security by judicious use of capabilities such as service accounts, network policies, authentication, authorization, admission control, AppArmor, and secrets.

Remember that a Kubernetes cluster is one part of a bigger system that includes other software systems, people, and processes. Kubernetes can't solve all problems. You should always keep in mind general security principles, such as defense in depth, a need-to-know basis, and the principle of least privilege. In addition, log everything you think may be useful in the event of an attack and have alerts for early detection when the system deviates from its state. It may be just...

Running a multi-tenant clusters

In this section, we will look briefly at the option to use a single cluster to host systems for multiple users or multiple user communities (which is also known as multi-tenancy). The idea is that those users are totally isolated and may not even be aware that they share the cluster with other users. Each user community will have its own resources, and there will be no communication between them (except maybe through public endpoints). The Kubernetes namespace concept is the ultimate expression of this idea. But, they don't provide absolute isolation. Another solution is to use virtual clusters where each namespace appears as a completely independent cluster to the users.

The case for a multi-tenant clusters

Why should you run a single cluster for multiple isolated users or deployments? Isn't it simpler to just have a dedicated cluster for each user? There are two main reasons: cost and operational complexity. If you have many relatively small...

Summary

In this chapter, we covered the many security challenges facing developers and administrators building systems and deploying applications on Kubernetes clusters. But we also explored the many security features and the flexible plugin-based security model that provides many ways to limit, control, and manage containers, pods, and nodes. Kubernetes already provides versatile solutions to most security challenges, and it will only get better as capabilities such as AppArmor and various plugins move from alpha/beta status to general availability. Finally, we considered how to use namespaces and virtual clusters to support multi-tenant communities or deployments in the same Kubernetes cluster.

In the next chapter, we will look in detail into many Kubernetes resources and concepts, and how to use them and combine them effectively. The Kubernetes object model is built on top of a solid foundation of a small number of generic concepts such as resources, manifests, and metadata. This empowers...

Reference

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritycontext-v1-core
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#securitycontext-v1-core
• https://kubernetes.io/docs/concepts/security/pod-security-admission/
• Pod security admission https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/2579-psp-replacement