Chapter 4. Logging Analytics with Kibana 5.0

The previous chapter showed how to use the Elastic stack for a business (logging) use case, which confirms that Elastic is not only a solution made for technical use cases, but rather a data platform that you can shape depending on your needs.

In the logging use case field, one of the most implemented within the technical domain is the web server logging use case. This chapter is a continuation of the previous one in the sense that we are dealing with logs, but addresses the problem from a different angle.

The goal here is first to understand the web logs use case, then to start importing both data in Elasticsearch, and dashboards in Kibana. We will go through the different visualizations available as part of the dashboard to see what key performance indicators can be extracted from the logs.

Finally, we'll ask our dashboard a question and deduce some more high-level considerations from the data, such as security or bandwidth insights.

Technical use case - Apache server logs

Apache and NGINX are the most used web servers in the world; there are billions of requests served by those servers out there, to internal networks as much as to external users. Most of the time, they are one of the first logic layers touched in a transaction, so from there, one can get a very precise view of what is going on in term of service usage.

In this chapter, we'll focus on the Apache server, and leverage the logs that the server generates during runtime to visualize user activity. The logs we are going to use were generated by a website (www.logstash.net) Apache web server. They were put together by Peter Kim and Christian Dahlqvist, two of my solutions architect colleagues at Elastic (https://github.com/elastic/elk-index-size-tests).

As mentioned in the introduction, this data can be approached and analyzed from different angles, and we will try to proceed to a security and a bandwidth analysis.

The first aims to detect suspicious behavior...

Summary

In this chapter, we have looked at how to use Kibana 5.0 in the context of technical logging use cases by diving into the analysis of Apache server logs. We have learned how to leverage visualizations for different purposes, such as bandwidth or security analysis. In the next chapter, we'll get into the domain of metrics analysis by first using Beats, the Elastic Stack data shipper.