Preface

Kibana is an open source data exploration and visualization platform. It is part of Elastic Stack, where we have Elasticsearch, Logstash, and Beats, along with Kibana. Using Kibana, we can explore data visually and can analyze it in real time. Kibana enables us to implement APM for application performance monitoring and Timelion enables us to play with time-series data. Then we have Dev Tools, by means of which we can run Elasticsearch queries direct from the Kibana interface. We have ML, by means of which we can predict future trends or ascertain anomalies in the data. Kibana provides us with Reporting, through which we can export CSV or PDF reports, Monitoring, to get insights into the complete Elastic Stack, and Watcher, to alert you in the event of any issue with the data.

Kibana, along with other Elastic Stack components, provides us with full-stack monitoring capability. Using Beats, we can get system metrics, log data, packet data, and so on. Logstash enables us to retrieve data from any other possible sources, including DBMS, CSV, or any other third-party tool, and then, using APM, we can fetch application data to monitor application performance. In this way, using Kibana, we can have an end-to-end monitoring system where a single dashboard can show all key performance indicators.

This book is there to help you understand the core concepts and the practical implementations, by means of which you can start using Kibana for a variety of use cases. It covers how to ingest data from different sources, using Beats or Logstash, into Elasticsearch, and then how to explore, analyze, and visualize it in Kibana. It covers how to play with time-series data to create complex graphs using Timelion and show them on your dashboard along with other visualizations, and then how to embed your dashboard or visualization on a web page. You will also learn about APM to monitor your application by installing and configuring the APM server and APM agents. We have also covered different X-Pack features, such as user and role management under security, alerting, monitoring, and ML. This book will also explain how to create ML jobs to find anomalies in your data.

Who this book is for

This book is for anybody who wants to explore data. We used to obtain data from different sources, which can be scattered. Using Kibana, we can arrange, analyze, and visualize it, and can then retrieve the relevant information from that diffuse data. For this book, no prior knowledge is required, and anyone can start working on Kibana using the simple introduction and practical implementations in the book. In this book, the focus is on a practical approach, where aspects are explained using practical examples, images, and a stepwise approach, where you need to sequentially follow a number of steps in order to achieve something. In this way, it is quite easy to understand the topics and you can easily implement the given steps

What this book covers

Chapter 1, Introducing Kibana, introduces Elastic Stack, where we explain the different components of Elastic Stack, including Elasticsearch, Logstash, Kibana, and different Beats. The introduction is followed by an explanation of the different use cases of Elastic Stack, including System Performance Monitoring, where we monitor system performance, Log Management, where we collect different logs and monitor them from a central location, Application Performance Monitoring, where we monitor our application by connecting it to a central APM server, Application Data Analysis, where we analyze the application data, Security Monitoring and Alerting, where we can secure our stack using X-Pack and monitor it regularly, while also being able to configure alerts to keep an eye on any change that may impact system performance, and finally Data Visualization, where we use Kibana to create different types of visualizations using available data.

Chapter 2, Getting Data into Kibana, covers different ways to get data in Elasticsearch. We examine how Beats can be installed on a server to send data, since they are lightweight data shippers. Under Beats, we cover Filebeat, for reading file data, including apache logs, system logs, and application logs, and can then send these logs to Elasticsearch directly or using Logstash. We configure Metricbeat to read system metrics, such as CPU usage, memory usage, MySQL metrics, and Packetbeat, by means of which we can read network packet data to glean insights from it. After that, we cover how Logstash can be used to get the data and apply filters before sending it to Elasticsearch.

In the first section, we cover how to fetch CSV data using Logstash, where we pass a CSV file as input and specify the columns to send the data to Elasticsearch. After that, we explain how to configure the JDBC plugin to fetch MySQL data by running the SQL statement and applying the tracking column, by means of which the incremental data can be fetched in Logstash. After reading the MySQL data, it is pushed to Elasticsearch for analysis. Using Beats and Logstash, we can push data into Elasticsearch but, in order to analyze and visualize the data, we need this data in Kibana and, for that, we have to create index patterns in Kibana. Once the index pattern is created, we can see the data under the Discover option in Kibana, where we can apply a filter, run queries, and select fields to display.

Chapter 3, Exploring Data, describes Kibana Discover, and how we can explore data using Discover. In the beginning, we cover how to discover your data by means of different options provided in Kibana Discover, including how to limit the number of fields to display in order to focus on the dataset, which is more relevant than the other not so relevant fields. Then, we discover how to expand a document display to check all available fields, along with the option to view surrounding documents and single documents. From this screen, we can also apply the filter to any field. Then, we cover different ways to dissect our data, including filtering the data by applying the time-based filter, filtering the data based on different document fields, and applying queries to your dataset. We then explore how to save the searched data so that this search data, along with filter options, can be available to us whenever we want to use them again. After saving the search data, we can also export it from Kibana and save it into a file that can later be imported back into Kibana.

Chapter 4, Visualizing Data, explains how to visualize the data once it is available in Kibana after creating the index pattern. We begin with basic charts, where we cover a number of chart creations, including the area chart, heat map, and pie chart. We also explain how we can transform one type of chart into other by taking the examples of the area chart, line chart, and bar chart in the same way that we can change a pie chart into a donut, or vice versa. After that, we delve into data tables, by means of which we can generate tabular visualizations of data in which we can add additional metrics columns, along with actual data columns. We then cover metric-type visualizations, where we can display some metric values and tag clouds, which can be used to display word clouds with a link to filter out the data accordingly.

Chapter 5, X-Pack with Machine Learning, explains how X-Pack adds additional features to the existing Elastic Stack setup. We begin with an introduction to X-Pack, followed by the X-Pack installation process. We then delve into the different features of XPack, such as security, by means of which we can secure our Elastic Stack. As regards security, we cover user and role management by creating users, and roles, and then assign roles to the users. Following on from security, we cover monitoring, from the perspective of both an overview and a detailed view, where we can see the search and indexing rate. We then cover alerting, where we configure watch to send alert notifications by email. Following on from alerting, we cover reporting, by means of which we can generate CSV or PDF reports and download them. Finally, we cover ML, by means of which we create single- and multi-metric jobs and analyze the data by finding the anomaly and predicting future trends.

Chapter 6, Monitoring Applications with APM, covers Elastic APM and explains how we can monitor an application. We begin with APM components, which are APM Agents, APM Servers, Elasticsearch, and Kibana. After that, we delve into each of them in detail. APM Agents are open source libraries that can be configured in any of the supported language/libraries. Currently, we have support for Django and flask frameworks for Python, Java, Go, Node.js, Rails, Rack, RUM - JS, and Go. We can configure them to send application metrics and errors to the APM Server. We then cover the APM Server, which is again an open source software written in Go. The principal task of the APM Server is to receive data from different APM Agents and send it to Elasticsearch Cluster. Elasticsearch takes the APM data, which can be viewed, searched, or analyzed in Elasticsearch. Once data is pushed in Elasticsearch, we can display it in Kibana using a dedicated APM UI or through the Kibana Dashboard.

Chapter 7, Kibana Advanced Tools, describes Timelion and Dev Tools, which are quite useful tools in Kibana. We begin with an introduction to Timelion, and then different functions that are available in Timelion, such as the .es() function to set the Elasticsearch data source, and its different parameters, such as index, metric, split, offset, fit, and time field. We then cover other functions, such as .static(), to create static lines on the x-axis, the .points() function to convert the graph into a point display, the .color() function to change the color of the plot, the .derivetive() function to plot the difference in value over time, the .label() function to set the label for data series, the .range() function to limit the graph display between a particular min and max range, and finally the .holt() function to forecast the future trend or to ascertain the anomaly in the data. For a complete reference of functions, we can refer to the help section in Timelion. We then cover the use cases of Timelion. After Timelion, we describe Dev Tools, by means of which we can do multiple things. After the introduction to Dev Tools, we cover different Dev Tools options, including Console, by means of which we can execute Elasticsearch queries and can get the response on the same page. We then examine the Search Profiler, through which we can profile any Elasticsearch query by getting the details of the query components. Finally, we look at Grok Debugger, where we can create the Grok Pattern to parse sample data, thereby enabling the unstructured sample data to be converted into structured data. This structured data can then be used for data analysis or visualization and suchlike.

To get the most out of this book

To get the most out of this book, no prior knowledge is required. Anyone who wants to analyze their data can use this book to learn how to do so.

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

1. Log in or register at www.packt.com.
2. Select the SUPPORT tab.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

• WinRAR/7-Zip for Windows
• Zipeg/iZip/UnRarX for Mac
• 7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Kibana-7-Quick-Start-Guide. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "We can add the cloud ID of Elasticsearch under the Elastic Cloud section in the metricbeat.yml file."

A block of code is set as follows:

input 
{
    file 
    {
        path => "/var/log/apache.log"
        type => "apache-access"  
        start_position => "beginning"
    }  
}
filter 
{
   grok 
   {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
   }
}
output 
{
    elasticsearch 
    {    
        hosts => ["localhost:9200"] 
    }
}

Any command-line input or output is written as follows:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/crimes.conf

Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: Select System info from the Administration panel.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [e-mail].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.