Auditing Using Falco, DevOps AI, and ECK

Bad people do bad things.

Good people do bad things.

Accidents happen.

The preceding statements have one thing in common: when any one of them occurs, you need to find out what happened and who did it.

Too often, auditing is considered only when we think of some form of attack. While we certainly require auditing to find "bad people," we also need to audit everyday standard system interactions.

Kubernetes includes logs for most of the important system events that you will need to audit, but it doesn't include everything. As we discussed in previous chapters, all API interactions will be logged by the system, which includes the majority of events you need to audit. However, there are tasks that users execute that will not go through the API server and may go undetected if you are relying on API logs for all of your auditing.

There are tools to address the gaps in the native logging functionality. Open source...

Technical requirements

To complete the exercises in this chapter, you will need to meet the following technical requirements:

• An Ubuntu 20.04 server with a minimum of 8 GB of RAM and at least 5 GB of free disk space for persistent volumes
• A KinD cluster installed using the instructions in Chapter 2, Deploying Kubernetes Using KinD
• Helm3 binary (you should already have this installed from Chapter 4, Services, Load Balancing, ExternalDNS, and Global Balancing)

You can access the code for this chapter by going to this book's GitHub repository: https://github.com/PacktPublishing/Kubernetes---An-Enterprise-Guide-2E/tree/main/chapter10.

Exploring auditing

In most environments where you run Kubernetes clusters, you will need to have an auditing system in place. While Kubernetes has some auditing features, they are often too limited for an enterprise to rely on for a complete audit trail, and logs are often only stored on each host filesystem.

In order to correlate events, you are required to pull all the logs you want to search through on your local system, and manually look through logs or pull them into a spreadsheet and attempt to create some macros to search and tie information together.

Fortunately, there are many third-party logging systems available for Kubernetes. Optional paid systems such as Splunk and Datadog are popular solutions and open source systems including the EFK stack are commonly used and included with many Kubernetes distributions. All of these systems include some form of a log forwarder that allows you to centralize your Kubernetes logs so you can create alerts, custom queries, and...

Introducing Falco

Falco is an open source system from Sysdig that adds anomaly detection functionality for pods in Kubernetes clusters. Out of the box, Falco includes a base set of powerful, community-created rules that can monitor a number of potentially malicious events, including the following:

• When a user attempts to modify a file under /etc
• When a user spawns a shell on a pod
• When a user stores sensitive information in a secret
• When a pod attempts to make a call to the Kubernetes API server
• Any attempts to modify a system ClusterRole
• Or any other custom rule you create to meet your needs

When Falco is running on a Kubernetes cluster it watches events, and based on a set of rules, it logs events on the Falco pod that can be picked up by a system such as Fluentd, which would then forward the event to an external logging system.

In this chapter, we will explain the configuration of Falco using the technical requirements for our...

Exploring Falco's configuration files

Before you install Falco, you need to understand the configuration options that are available, and that starts with the initial configuration file that will be used to configure how Falco creates events.

The Falco project includes a set of base configuration files that you can use for your initial auditing. It is highly likely that you will want to change the base configuration to fit your specific enterprise requirements. In this section, we will go over a Falco deployment and provide a basic understanding of the configuration files.

Falco is a powerful system that can be customized to fit almost any requirement you may have for security. Since it is so extensible, it's not possible to cover every detail of the configuration in a single chapter, but like many popular projects, there is an active GitHub community at https://github.com/falcosecurity/falco where you can post issues or join their Slack channel.

Recent versions...

Deploying Falco

We have included a script to deploy Falco, called install-falco.sh, in the GitHub repository in the chapter10 folder.

The two most popular methods of deploying Falco to a Kubernetes cluster are using the official Helm chart or a DaemonSet manifest from the Falco repo. For the purposes of this module, we will deploy Falco using Helm and a custom values.yaml from the book's GitHub repository.

To deploy Falco using the included script, execute the script from within the chapter10 folder by executing ./install-falco.sh.

The steps that the script performs are detailed in the following list and will be explained in additional detail in this section.

The script executes the following tasks:

1. Creates the Falco namespace
2. Adds the charts from the falcosecurity.github.io/charts repository
3. Installs Falco using Helm in the falco namespace using the custom values file along with the custom rules file for our NGINX example

Falco...

Introducing DevOPs AI

Our function isn't really an AI function, it's a "dumb" function that only reacts to a certain event and it has no logic to understand if the event should be allowed or not. Our example is only meant to introduce you to using Falcosidekick to increase your security by forwarding events to an external system.

There are no limits to what you can create once events are forwarded – you can forward events to GCP Pub/Sub, and once an event is in GCP you can leverage any of Google's tools to create complex decisions that leverage any of the Google AI tools.

Obviously, AI is an entire series of books by itself, so we decided to stick to a single, static use case. While it's true that AI is much cooler than static checks, you can still create an effective automatic response engine using standard functions.

Now to see this in action!

Understand automatic responses to events

One of the best methods to understand...

Summary

This chapter covered how to create an enhanced auditing system for your Kubernetes cluster. We started the chapter by introducing Falco, an auditing add-on that was donated to the CNCF by Sysdig. Falco adds a level of auditing that Kubernetes does not include, and combined with the included auditing functionality, provides an audit trail for everything from API access to actions in a pod.

We explained how Falcosidekick can be used to forward events to other systems to perform complex tasks like creating automated response engines using systems like Kubeless to execute functions based on certain events. This was just a small example, but the possibilities are endless with the integrations that Falcosidekick includes, including Pub/Sub, Cloud Run, Lambda, and more.

Logs aren't beneficial if you can't store them in a logging system that allows you to store logs on persistent storage and offers a management interface to search logs and create dashboards. We installed...

Questions

1. If you need to edit an included Falco rule, which file would you edit?
   1. falco.yaml
   2. falco_rules.yaml
   3. falco_rules.changes.yaml
   4. falco_rules.local.yaml
2. Which of the following is a common log forwarder used by Kubernetes?
   1. Kube-forwarder
   2. Fluentd
   3. Forwarder
   4. Kubernetes doesn't use forwarders
3. What is the product that provides a way to present logs using visualizations and dashboards when you deploy the EFK stack?
   1. Fluentd
   2. Elasticsearch
   3. Kibana
   4. Excel
4. Which of the following tools forwards only Falco logs to a central logging system?
   1. Falco
   2. Falcosidekick
   3. The Kubernetes API server
   4. All products forward every log, not just the Falco logs
5. What is the name of the object in Falco that allows you to create a collection of items?
   ...