Exploring Logs with Grafana Loki

In this final chapter of Section 2: Real-World Grafana, we're going to shift gears a bit. So far, we've been operating under a dashboard-oriented paradigm in terms of how we use Grafana. This is not too unusual since Grafana has always been structured around the dashboard metaphor. Now, the development of Explore in Grafana 6 brings an alternative workflow – one that is data-driven and, dare I say it, exploratory.

Grafana really shines when working with numerical and some forms of textual data, but what if the data includes substantial amounts of log data? Every day, countless applications disgorge not only standard numerical metrics but also copious text logs. If you've ever enabled debug mode in an application, then you've seen how quickly a few meager kilobytes of information can quickly become a flood of gigabytes worth of repetitive, inscrutable gibberish...

Loading system logs into Loki

To get started, cd to the ch10 directory in your clone of this book's repository.

Our first step is to download and launch the Loki pipeline services with Docker Compose. We will use a sample docker-compose.yml file, which can be downloaded from the Loki GitHub repository (found at https://github.com/grafana/loki). By now, the docker-compose.yml file should seem familiar and pretty straightforward. In our initial deployment, we will set up three services: loki, promtail, and grafana. Let's have a quick look at the configuration for each service:

  loki:
    image: "grafana/loki:${LOKI_TAG-latest}"
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki
    volumes:
      - "${PWD-.}/loki:/loki"

First up is the Loki service itself. Loki will provide the log storage service that the data source will access to search...

Visualizing Loki log data with Explore

Go to Explore and make sure Loki is set as the data source.

Welcome to Explore! Things look quite a bit different from what you may remember from using Explore with other data sources. Let's take a quick tour of some of the basic UI features:

The features highlighted in the preceding screenshot are as follows:

1. Log labels/Metrics Explorer: This is where you can use the Log labels or Metrics Explorer to access log labels as they are generated or parsed from your logs. Logs can also be aggregated into metrics. Loki is architected somewhat differently than Elasticsearch in that it doesn't create a database of indexed log files. Rather, it simply associates logs with text labels, similar to metrics labels in Prometheus.
2. Query field: You enter log stream queries here using the LogQL query language, which is very similar to PromQL for Prometheus. Selections made...

Adding additional service logs

We'd like to get Loki to not only aggregate the system logs, but also the logs our services are generating. We could adjust the Promtail configuration to look for the logs wherever they're stored by Docker, but they could be tricky to find and the container may not sync them with the filesystem in a timely enough fashion.

Luckily, the folks at Grafana have provided a log driver for Docker that can deliver logs to Loki directly, thus bypassing Promtail entirely. It requires downloading a special Loki log driver and requires adjusting the docker-compose.yml file so that it includes the new logging information. To download and install the driver, run the following command:

          % docker plugin install grafana/loki-docker-driver:latest --alias loki
          --grant-all-permissions
        

To confirm the installation, run the following command:

% docker plugin ls

ID NAME DESCRIPTION ENABLED
692bec0b6ade loki...

Querying logs and metrics with Explore

Adding Prometheus to the mix is relatively straightforward. We'll add a new Prometheus service while sending its logs to Loki to be aggregated (why not?). We'll also need to configure Prometheus to scrape the metric endpoints of our services. We did this earlier in this book, so it should be no problem for us to configure scrapers for each service.

First, let's add Prometheus to our stack:

  prometheus:
    image: "prom/prometheus:${PROM_TAG-latest}"
    ports:
      - "9090:9090"
    volumes:
      - "${PWD-.}/prometheus:/etc/prometheus"
    command: --config.file=/etc/prometheus/prometheus.yaml
    networks:
      - loki
    logging:
      driver: loki
      options:
        loki-url: "http://host.docker.internal:3100/loki/api/v1/push"

We'll need to create a volume directory for Prometheus to store our configuration file (namedprometheus.yml).

The...

Summary

We've reached the end of Chapter 10, Exploring Logs with Grafana's Loki. In this chapter, we learned how to use Explore with the Loki data source to perform ad hoc analysis of logs and aggregated log metrics and we deployed a Loki pipeline to aggregate filesystem log files and, by using a custom driver, the logs generated by Docker containers. Then, we used Prometheus to collect dozens of metrics about those container services. Finally, using the Split feature, we made side-by-side comparisons of both the log and service metrics.

With that, we've also reached the end of Section 2: Real-World Grafana. In Section 3: Managing Grafana, we'll step out of our role as an end user of Grafana and into that of an administrator. We'll learn about how to manage dashboards, users, and teams. We'll also look at how to secure the Grafana server by authenticating our users with services such as OAuth2 and LDAP. Finally, we'll explore the rapidly...