12

Cloud Logging

Logging provides visibility into your environment and aids in troubleshooting and incident response. In this chapter, we will discuss what Cloud Logging is, how it works, the different types of logs and their applications, and how to collect, store, analyze, and export logs. We will also look at who log producers and consumers are and how to export logs to a centralized logging solution or a Security Information and Event Management (SIEM) system running either on-premises or in the cloud. Finally, we will discuss how to securely store and keep logs to meet regulatory requirements.

In this chapter, we will cover the following topics:

• Overview of Google Cloud Logging
• Understanding log categories
• Log management
• Logging and auditing best practices

Introduction to Google Cloud logging

Cloud Logging is a managed service on Google Cloud. It gives you the ability to collect, store, search, analyze, monitor, and set alerts based on logs that are collected. You can use Cloud Logging to collect log data from over 150 applications, Google Cloud components, third-party cloud providers, and any combination of cloud platforms (hybrid environments). Cloud Logging, formerly known as Stackdriver, is part of Google Cloud’s operations suite. It includes a console interface called the Logs Explorer, query logs, and an API to manage logs programmatically.

Figure 12.1 – Google Cloud’s operations suite

Cloud Logging is one of the four components of Google Cloud’s operations suite, as shown in Figure 12.1. Let’s explore each of these components in more depth. Cloud Logging captures all logs and provides searching and filtering capabilities. Cloud Logging derives metrics from logs,...

Log categories

In this section, we broadly divide logs into three categories. We will look at each category from the perspective of security.

Figure 12.3 – Log categories

Figure 12.3 illustrates the different types of log categories: security logs, user logs, and platform logs. Security logs consist of admin activity logs, data access logs, system event logs, and transparency logs. User logs are generated by user software, services, or applications and are written to Cloud Logging using a logging agent, the Cloud Logging API, or the Cloud Logging client libraries. Google Cloud Platform logs are service-specific logs that can help you better understand the Google Cloud services you’re using. VPC flow logs, firewall logs, and other API logs are examples of platform logs.

Our focus will be only on logs that are useful from a security perspective. That doesn’t necessarily mean that we will only look at the security logs category. We...

Log management

A key component of logging is the ability to export logs so they can be consumed by a service running on the cloud or by a third-party solution. You will also learn how to aggregate logs, build log pipelines, perform log analysis, and deliver logs to SIEM solutions.

Log producers

All resources that generate logs, such as products, services, and applications, whether they are in the cloud or a hybrid environment, are classified as log producers. These could be Google Cloud services or services running in your on-premises data center or a third-party cloud service provider.

Log consumers

Log consumers are services such as centralized logging solutions and SIEM solutions. Any service that has the ability to consume logs and provide functions such as alerting, troubleshooting applications, and business intelligence is considered a log consumer. Cloud Logging on Google Cloud is an example of such a service.

Log Router

The term Log Router has been used a...

Logging and auditing best practices

In this last section, we will cover some logging and auditing best practices. From an exam perspective, you are not really tested with questions on these practices, but you may find indirect references to them. Since Data Access audit logs are not enabled by default, it’s important that you review and enable Data Access audit logs for relevant services.

Note

You can find more details on how to enable these logs at https://packt.link/w0ZWY.

Let’s look at some best practices for logging and auditing:

• Ensure that you specify the correct log storage destination—Google Cloud Storage for archival and retention and BigQuery for analytics and forensics
• Configure appropriate permissions for the access of your logs
• Ensure that sensitive data is stripped from the logs before they are stored
• Periodically review your audit logs that are stored in Cloud Logging, BigQuery, or your third-party SIEM solutions

...

Summary

In this chapter, we looked at the different categories of logs and what types of logs are available in each of those categories. We also looked at key properties of the different types of logs, such as whether they are enabled by default and the default retention period associated with each category. We also looked at the key components of Cloud Logging, such as the Cloud Logging API, the Log Router, log producers, and log consumers. Then, we looked at how to create log exports, how to specify the filters on what you want to exclude or include, and the different destinations that are available to store your logs. We covered the use of logging pipelines for analytics and storage. Finally, we looked at how to address compliance needs through logging and discussed best practices for logging and auditing.

In the next chapter, we will learn about Cloud Monitoring. That chapter will build on top of the knowledge gained in this one and introduce the topics of log-based metrics...

Further reading

For more information on logging, refer to the following links:

• Design patterns for exporting Cloud Logging data: https://packt.link/glqqx
• Logging, Monitoring, and Observability in Google Cloud: https://packt.link/1EnJt
• Exporting Cloud Logging data for security and analytics: https://packt.link/mF2Tg
• Exporting Cloud Logging data to Elasticsearch: https://packt.link/dvg2g
• Exporting logging data to Splunk: https://packt.link/t9ipu
• Using BigQuery and Cloud Logging to analyze BigQuery usage: https://packt.link/tmCSd
• Monitor and Log with Google Cloud Operations Suite: https://packt.link/0v0dh