Query logging and limits

Depending on your environment and use case, you may end up in a situation in which you are experiencing performance problems due to expensive queries being run or a high number of queries. Prometheus provides built-in ways to log completed queries, maintain a log of active queries, and set limits to prevent overly broad queries from pulling in too much data. Using these, we can both determine the source of issues and put safeguards in place to prevent them from happening.

Query logging

Query logging can be configured via the Prometheus configuration file under the global key with the query_log_file setting. This setting accepts a file path to which Prometheus will write all queries that it completes:

global:
    query_log_file: /var/log/prometheus/my_prometheus_queries.log

Each entry in the query log file includes helpful information such as the timestamp of the query, the query itself, and other useful statistics about the query...