You're reading from Kubernetes - A Complete DevOps Cookbook

Product typeBook

Published inMar 2020

PublisherPackt

ISBN-139781838828042

Edition1st Edition

Tools

Kubernetes

Concepts

DevOps

Author (1)

Murat Karslioglu

Logging with Kubernetes

In this chapter, we will discuss cluster logging for Kubernetes clusters. We will talk about setting up a cluster to ingest logs, as well as how to view them using both self-managed and hosted solutions.

In this chapter, we will cover the following recipes:

Accessing Kubernetes logs locally
Accessing application-specific logs
Building centralized logging in Kubernetes using the EFK stack
Logging with Kubernetes using Google Stackdriver
Using a managed Kubernetes logging service
Logging for your Jenkins CI/CD environment

Technical requirements

The recipes in this chapter expect you to have a functional Kubernetes cluster deployed by following one of the recommended methods described in Chapter 1, Building Production-Ready Kubernetes Clusters.

The Logging for your Jenkins CI/CD environment recipe in this chapter expects you to have a functional Jenkins server with an existing CI pipeline created by following one of the recommended methods described in Chapter 3, Building CI/CD Pipelines.

The Kubernetes command-line tool kubectl will be used for the rest of the recipes in this chapter since it's the main command-line interface for running commands against Kubernetes clusters. We will also use helm where Helm charts are available in order to deploy solutions.

Accessing Kubernetes logs locally

In Kubernetes, logs can be used for debugging and monitoring activities to a certain level. Basic logging can be used to detect configuration problems, but for cluster-level logging, an external backend is required to store and query logs. Cluster-level logging will be covered in the Building centralized logging in Kubernetes using the EFK stack and Logging Kubernetes using Google Stackdriver recipes.

In this section, we will learn how to access basic logs based on the options that are available in Kubernetes.

Getting ready

Clone the k8sdevopscookbook/src repository to your workstation to use the manifest files in the chapter10 directory, as follows:

$ git clone https://github.com/k8sdevopscookbook/src.git
$ cd src/chapter10

Make sure you have a Kubernetes cluster ready and kubectl and helm configured to manage the cluster resources.

How to do it…

This section is further divided into the following subsections to make this process easier:

Accessing...

Accessing application-specific logs

In Kubernetes, pod and deployment logs that are related to how pods and containers are scheduled can be accessed through the kubectl logs command, but not all application logs and commands are exposed through Kubernetes APIs. Getting access to these logs and shell commands inside a container may be required.

In this section, we will learn how to access a container shell, extract logs, and update binaries for troubleshooting.

Getting ready

Clone the k8sdevopscookbook/src repository to your workstation to use the manifest files under the chapter10 directory, as follows:

$ git clone https://github.com/k8sdevopscookbook/src.git
$ cd src/chapter10

Make sure you have a Kubernetes cluster ready and kubectl configured to manage the cluster resources.

How to do it…

This section is further divided into the following subsections to make this process easier:

Getting shell access in a container
Accessing PostgreSQL logs inside a container

Getting shell access...

Building centralized logging in Kubernetes using the EFK stack

As described in the the Accessing Kubernetes logs locally section, basic logging can be used to detect configuration problems, but for cluster-level logging, an external backend is required to store and query logs. A cluster-level logging stack can help you quickly sort through and analyze the high volume of production log data that's produced by your application in the Kubernetes cluster. One of the most popular centralized logging solutions in the Kubernetes ecosystem is the Elasticsearch, Logstash, and Kibana (ELK) stack.

In the ELK stack, Logstash is used as the log collector. Logstash uses slightly more memory than Fluent Bit, which is a low-footprint version of Fluentd. Therefore, in this recipe, we will use the Elasticsearch, Fluent-bit, and Kibana (EFK) stack. If you have an application that has Logstash dependencies, you can always replace Fluentd/Fluent Bit with Logstash.

In this section, we will learn how...

Logging Kubernetes using Google Stackdriver

In this section, we will use Google Stackdriver Kubernetes Engine Monitoring to monitor, isolate, and diagnose our containerized applications and microservices environments. You will learn how to use Stackdriver Kubernetes Engine Monitoring to aggregate logs, events, and metrics from your Kubernetes environment on GKE to help you understand your application's behavior in production.

Getting ready

Make sure you have a Google Kubernetes Engine (GKE) cluster ready and kubectl configured to manage the cluster resources. If you don't have one, you can follow the instructions in Chapter 1, Building Production-Ready Kubernetes Clusters, in the Configuring a Kubernetes cluster on Google Cloud Platform recipe.

How to do it…

This section is further divided into the following subsections to make this process easier:

Installing Stackdriver Kubernetes Engine Monitoring support for GKE
Configuring a workspace on Stackdriver
Viewing GKE...

Using a managed Kubernetes logging service

Running an EFK stack to store and maintain Kubernetes logs in your cluster is useful until something goes wrong with your cluster. It is recommended that you keep your log management system and production cluster separate so that you have access in case of cluster failure.

In this section, we will learn how to use some of the freely available SaaS solutions to keep your cluster logs accessible, even if your cluster is not available.

Getting ready

Make sure you have a Kubernetes cluster ready and kubectl configured to manage the cluster resources.

How to do it…

This section is further divided into the following subsections to make this process easier:

Adding clusters to Director Online
Accessing logs using Director Online

Connecting clusters to Director Online

OpenEBS Director provides a freely managed EFK stack as a SaaS solution so that you can store and manage your Kubernetes cluster logs. In this recipe, we will add our Kubernetes...

Logging for your Jenkins CI/CD environment

CI/CD pipelines can generate a great amount of metadata every day in busy build environments. Elasticsearch is the perfect platform for feeding this kind of data from Jenkins.

In this section, we will learn how to enable and access the logs of our Jenkins instance and analyze team efficiency.

Getting ready

All the operations mentioned in this recipe require a fully functional Jenkins deployment, as described in Chapter 3, Building CI/CD Pipelines, in the Setting up a CI/CD pipeline in Jenkins X section.

Clone the k8sdevopscookbook/src repository to your workstation to use the manifest files in the chapter10 directory:

$ git clone https://github.com/k8sdevopscookbook/src.git
$ cd src/chapter10

Make sure you have a Kubernetes cluster, Jenkins X, and an EFK stack ready and that kubectl has been configured so that you can manage the cluster resources.

How to do it…

This section will show you how to feed Jenkins logs to Elasticsearch. This section...

The rest of the chapter is locked

You have been reading a chapter from

Kubernetes - A Complete DevOps Cookbook

Published in: Mar 2020Publisher: PacktISBN-13: 9781838828042

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Author (1)

Murat Karslioglu

Murat Karslioglu is a distinguished technologist with years of experience using infrastructure tools and technologies. Murat is currently the VP of products at MayaData, a start-up that builds data agility platform for stateful applications, and a maintainer of open source projects, namely OpenEBS and Litmus. In his free time, Murat is busy writing practical articles about DevOps best practices, CI/CD, Kubernetes, and running stateful applications on popular Kubernetes platforms on his blog, Containerized Me. Murat also runs a cloud-native news curator site, The Containerized Today, where he regularly publishes updates on the Kubernetes ecosystem.
Read more about Murat Karslioglu

Other recommended products

Related to this chapter

kubectl: Command-Line Kubernetes in a Nutshell

Kubernetes is a de facto container orchestration system. To manage Kubernetes, you must understand how to work with kubectl, its command-line tool that lets you interact with your clusters to deploy applications and manage their lifecycle. This book is a comprehensive introduction for those who are new to Kubernetes management via the command line.

BookNov 2020136 pages

Kubernetes in Production Best Practices

Kubernetes is a portable, extensible, and open source platform used for orchestrating and managing workloads and services. This book provides detailed coverage of cloud industry standards and best practices and serves as a reference guide for building and operating production-ready Kubernetes infrastructure.

BookMar 2021292 pages

Kubernetes Cookbook

Kubernetes is one of the most popular, sophisticated, and fast-evolving container orchestrators. In this book, you’ll learn the essentials and find out about the advanced administration in Kubernetes. We’ll take you through a step-by-step hands-on approach, which will familiarize you with the Kubernetes ecosystem.

BookMay 2018554 pages

Mastering Service Mesh

Service Mesh helps overcome the operational challenges of connecting, securing, controlling, and observing modern microservices deployment. This book shows you exactly how to use a Service Mesh architecture to manage and operationalize your microservices-based applications.

BookMar 2020626 pages

Hands-On Kubernetes on Azure

This book will help readers to Deploy web applications securely in Microsoft Azure with docker container and having the need for clustering services to achieve high availability, dynamic scalability, and to monitor applications

BookMar 2019258 pages

The DevOps 2.3 Toolkit

Viktor Farcic’s latest book, The DevOps 2.3 Toolkit: Kubernetes, will take you on a hands-on journey with Viktor into the world of Kubernetes, and the tools not only behind the official project but also the wide-range of third-party apps that are available for you to use.

BookSep 2018418 pages

Cloud Native with Kubernetes

Cloud Native with Kubernetes will guide you effectively - from your first steps using Kubernetes right up to running enterprise-grade cloud native applications with best practices. The book covers every aspect of deploying, securing, and operating modern cloud native applications on Kubernetes.

BookJan 2021446 pages

The DevOps 2.4 Toolkit

The DevOps Toolkit 2.4 is a deep exploration of continuous delivery and deployment in Kubernetes using Jenkins. It shows readers how to build, test, and deploy applications in Kubernetes using fully automated Jenkins pipelines.

BookNov 2019398 pages

Learn Kubernetes Security

Kubernetes is a widely used opensource orchestration platform for managing container applications. Developers are not yet aware of the pitfalls of a containerized environment. This book takes you through core security principles, best practices, and real-world use cases to learn mitigation or prevention strategies from known attacks and CVEs.

BookJul 2020330 pages

DevOps with Kubernetes

This book will guide you from container basic concepts to orchestrating containerized applications in Kubernetes. You’ll learn about the Kubernetes basic architecture, components, resources, admission control, and extensions. We’ll show you how to utilize Kubernetes services in three top cloud providers.

BookJan 2019484 pages

Kubernetes Design Patterns and Extensions

Kubernetes is an open source platform that automates scaling, deployment, and management of container-based applications. The book starts by explaining basics about container orchestration and then moves on to explain how the components of a complex system, such as Kubernetes, interact with each other and helps develop vital skills for troubleshooting Kubernetes clusters and applications running in Kubernetes clusters. Once you are done reading the book, you can easily develop and deploy your own applications with Kubernetes in no time.

BookSep 2018106 pages

Kubernetes for Developers

Kubernetes provides a means to describe what your application needs and how it should run by orchestrating containers on your behalf to operate your software across a single, dozens, or hundreds of machines. Originally created and open sourced from Google, you can use their technology to help run, test, and scale your applications regardless of which cloud provider you use, or even running it locally on your own hardware. This book will show you how to take advantage of this amazing technology by introducing the key concepts and walking you through examples in both NodeJS and Python.

BookApr 2018374 pages

Personalised recommendations for you

Based on your interests and search pattern

Designing and Implementing Microsoft Azure Networking Solutions

Designing and Implementing Microsoft Azure Networking Solutions Exam Ref AZ-700 is an all-encompassing guide to the AZ-700 exam and contains all the information you need to succeed in the world of virtual networking with Azure. With this book, you will be fully prepared for the exam and the world of cloud networking.

BookAug 2023524 pages

Microsoft 365 Security, Compliance, and Identity Administration

The Microsoft 365 Security, Compliance, and Identity Administration is a comprehensive guide that helps you employ Microsoft 365's robust suite of features and empowers you to optimize your administrative tasks.

BookAug 2023630 pages

Zero Trust Overview and Playbook Introduction

Get started on Zero Trust with this step-by-step playbook and learn everything you need to know for a successful Zero Trust journey with tailored guidance for every role, covering strategy, operations, architecture, implementation, and measuring success. This book will become an indispensable reference for everyone in your organization.

BookOct 2023240 pages

The Self-Taught Cloud Computing Engineer

This self-study book helps you master multiple clouds, including AWS, Azure, and GCP, and serves as a roadmap to becoming a certified cloud computing expert. The book will guide you to develop a professional cloud career by helping you build a broad cloud knowledge base, developing hands-on cloud computing skills, and getting cloud certified.

BookSep 2023472 pages

Technology Operating Models for Cloud and Edge

This book will help you build and create ownership of a technology operating model, as well as connect your leadership with engineering and operations, keeping your internal and external customers in mind. It provides practical tips on why, where, and how to make the cloud and edge platform paradigm sing for you, your team, and your organization.

BookAug 2023228 pages

Azure Architecture Explained

Azure is the preferred platform to build mission-critical and secure apps. This book provides comprehensive coverage of essential Azure products, services, and solutions vital for every solution architect's success. Elevate your knowledge and master the critical components of Azure to excel in your role with Azure Architecture Explained.

BookSep 2023446 pages

Pentesting Active Directory and Windows-based Infrastructure

This practical guide helps you explore the pentesting of Microsoft infrastructure in detail, and enhances your offensive skillset by showing you the different ways to perform security assessment. This book will help blue teamers and IT engineers get up to speed with possible security issues they may encounter in their Windows environments.

BookNov 2023360 pages

Practical Ansible

In Practical Ansible, you'll work with the latest release of Ansible and learn to solve complex issues quickly with the help of task-oriented scenarios. You'll start by installing and configuring Ansible to automate monotonous and repetitive IT tasks and get to grips with concepts such as playbooks, inventories, plugins, collections, and network modules.

BookSep 2023420 pages

Windows 11 for Enterprise Administrators

Microsoft’s launch of Windows 11 is a step toward satisfying the enterprise administrator’s needs for better management and enhanced user experience customization. This book provides the enterprise administrator with the knowledge needed to fully utilize the advanced feature set of Windows 11 Enterprise.

BookOct 2023286 pages

The Linux DevOps Handbook

This book is for software and IT professionals seeking knowledge on Linux systems and DevOps practices. This book will provide you with guidance and tools to learn and gain proficiency in managing Linux-based infrastructures and knowledge of DevOps.

BookNov 2023428 pages2