WIUNIX ACADEMY
Prometheus high cardinality: find the source and test a useful recording rule
Use TSDB statistics and bounded queries to locate series growth, fix instrumentation safely and validate aggregation without confusing cheaper queries with fewer stored series.
Make this guide yoursYOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment4 editable values
THE OUTCOME
What will you have at the end?
Identify a high-cardinality metric/label source and validate one counter recording rule while preserving diagnostic meaning.
Supported operating systems and versions
Documented Prometheus3-style stable APIs and promtool workflow. No server query/load/rule reload was executed during authoring.
Before you begin
- Existing protected Prometheus and matching promtool
- Read-only local API access and ownership of the selected application metric contract
- Existing source-controlled rule deployment and rollback procedure
LET’S BUILD
Prepare the selected operator host
These are operator-host packages only. Review the transaction before approving; do not install tools inside application containers. The cluster, server and application binaries must already exist with owner-approved versions. Use Bash for the following panels. Package installation changes this selected host but starts no new application server.
Use the existing Prometheus and compatible promtool. This path installs neither server nor promtool and preserves the installed rule paths/reload mechanism.
Selected operator host; approved package transaction
sudo apt-get update
sudo apt-get install --no-install-recommends ca-certificates curl bash coreutils
cat /etc/os-release
command -v bash curl timeout
'/usr/local/bin/promtool' --versionThe OS/version matches the selection, required tools resolve, and existing application/client versions satisfy the stated compatibility boundary. Stop if a binary is missing or incompatible.
Preserve host security and existing paths
Ubuntu/Debian may use AppArmor and host firewall rules. This workflow opens no inbound host port and changes no security profile. For denied file/network access, inspect the owning service logs and policy with its administrator. Keep existing executable/config paths; public path fields are reviewed example locations, not discovered installation paths.
Separate cardinality, churn and expensive queries
A time series is a metric name plus its complete label set. Request IDs, user IDs or raw URLs can create unbounded combinations. Cardinality counts active combinations; churn is the rate at which combinations appear/disappear. A slow query can also result from a broad time range or expensive expression even when series counts are stable.
Recording rules materialize derived series. They can reduce repeated query work but add stored series; they do not remove the original input series. Fix the source of uncontrolled labels before presenting aggregation as a storage cure. This guide assumes an existing protected Prometheus, not a new monitoring stack.
LET’S BUILD
Record the exact server and rule paths
Use the protected loopback endpoint on the Prometheus host. Authentication/TLS proxies require your existing approved client configuration rather than putting credentials into these public fields. Server and promtool should be compatible. Record the existing rule_files entries and reload procedure privately; the sample file is not active just because it exists.
Operator terminal; authorized lab context
'/usr/local/bin/promtool' --version
curl --noproxy '*' --fail --silent --show-error --max-time 5 'http://127.0.0.1:9090/api/v1/status/buildinfo'
curl --noproxy '*' --fail --silent --show-error --max-time 5 'http://127.0.0.1:9090/api/v1/status/flags'The intended local server/version and compatible validation tool are confirmed.
LET’S BUILD
Start with bounded TSDB statistics
The TSDB status endpoint exposes head series, top metric counts and label-value statistics. It is a diagnostic view of head data, not exact per-metric RSS or whole-retention disk accounting. Sample it at comparable times and correlate with deployment changes. Even diagnostic endpoints consume resources; avoid tight polling during an incident.
Operator terminal; authorized lab context
curl --noproxy '*' --fail --silent --show-error --max-time 10 'http://127.0.0.1:9090/api/v1/status/tsdb?limit=10'
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=prometheus_tsdb_head_series'A baseline head count and bounded top offenders are captured without exporting all label sets.
LET’S BUILD
Narrow one job and one metric before exploration
These queries assume the owned app exports the conventional HTTP counter shown. Adapt the metric contract with its owner if not; absent data is not zero traffic. Avoid an unbounded /series export or whole-server count-by query on an already overloaded server. Topk limits returned rows, not necessarily the engine work needed to compute them.
The final series-created counter is global Prometheus ingestion churn, not churn attributed to the selected application job. Its job/instance labels identify the Prometheus monitoring target, so do not interpret them as the offending app.
Operator terminal; authorized lab context
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=count(http_requests_total{job="api"})'
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=topk(10,count by (route) (http_requests_total{job="api"}))'
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=rate(prometheus_tsdb_head_series_created_total[5m])'Identify a concrete suspect route/label and whether active series or creation rate is growing.