WIUNIX ACADEMY
Prometheus high cardinality: find the source and test a useful recording rule
Page 2 of 3. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment4 editable values
Repair labels at instrumentation time
Prefer bounded route templates such as /items/:id, fixed outcome classes and a reviewed allowlist of dimensions. Keep user/request identity in a properly governed log/trace system when needed, not a metric label. Estimate combinations as a product of label values, then measure the actual joint combinations; not all combinations necessarily exist.
Removing a label at scrape time can collapse two samples into the same final label set, causing duplicate/out-of-order problems instead of correct aggregation. metric_relabel_configs labeldrop is not sum. Dropping an entire reviewed unwanted metric can be appropriate, but audit alert/dashboard dependencies first. Preserve useful status/route distinction when it explains an SLO.
LET’S BUILD
Create a private candidate rule and test file
Use a new directory. The next panels are file contents, not shell commands. The rule keeps job, route and status; it deliberately drops instance after computing a per-series counter rate. Applying rate before summing preserves per-instance counter reset handling.
Operator terminal; authorized lab context
umask 077
mkdir '/tmp/academy-prom-rules' && cd '/tmp/academy-prom-rules' || exit 1
(set -C; : > rules.yaml; : > rules-test.yaml) || exit 1
${EDITOR:-vi} rules.yaml rules-test.yamlTwo new candidate files exist without changing the active rule directory.
LET’S BUILD
Materialize only the reviewed aggregate
The group limit prevents silently creating unlimited output series; reaching it is an error to investigate, not automatic safe truncation. The output name describes retained labels and operation. Do not reuse an existing recording name with different semantics. The request counter and its route/status labels must be real in your application.
File contents; save at the displayed filename
groups:
- name: academy-http
interval: 1m
limit: 1000
rules:
- record: job_route_status:http_requests:rate5m
expr: sum by (job, route, status) (rate(http_requests_total{job="api"}[5m]))
The rule adds one bounded family of derived series; raw series remain until normal retention removes them.
LET’S BUILD
Test two counters and the exact retained labels
The fixture uses two instances with rates1 and2 per second over a stable five-minute window. The expected aggregate is3, and instance must be absent from the result. This proves this arithmetic case only; add your actual reset/staleness cases before production use.
File contents; save at the displayed filename
rule_files: [rules.yaml]
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'http_requests_total{job="api",route="/items",status="200",instance="a"}'
values: '0 60 120 180 240 300'
- series: 'http_requests_total{job="api",route="/items",status="200",instance="b"}'
values: '0 120 240 360 480 600'
promql_expr_test:
- expr: 'job_route_status:http_requests:rate5m{job="api"}'
eval_time: 5m
exp_samples:
- labels: 'job_route_status:http_requests:rate5m{job="api",route="/items",status="200"}'
value: 3
The fixture expects the aggregate value3 with exactly job/route/status labels.
LET’S BUILD
Validate syntax and evaluate the fixture
Run the installed compatible promtool in the candidate directory so relative rule_files resolves correctly. A syntax pass alone is not a rule-behavior test. A passing fixture is not a server load test or evidence the app exposes the assumed counter. Do not label these documented commands as author-executed tests.
Operator terminal; authorized lab context
cd '/tmp/academy-prom-rules' || exit 1
'/usr/local/bin/promtool' check rules rules.yaml
'/usr/local/bin/promtool' test rules rules-test.yamlBoth commands exit successfully; the fixture returns the exact value and label set.
LET’S BUILD
Integrate through the existing reload mechanism
Review the candidate, reference it in the owning source’s rule_files and validate the whole intended configuration through the existing deployment mechanism. Do not enable the lifecycle HTTP endpoint or restart the server simply for this guide. After the owner’s normal reload, inspect rule health/evaluation errors and compare raw expression to recording output over matching timestamps. Rule evaluation introduces its own cadence and initial empty window.
Operator terminal; authorized lab context
curl --noproxy '*' --fail --silent --show-error --max-time 10 'http://127.0.0.1:9090/api/v1/rules?type=record'
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=job_route_status:http_requests:rate5m{job="api"}'
curl --noproxy '*' --fail --silent --show-error --get --max-time 10 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=sum by(job,route,status)(rate(http_requests_total{job="api"}[5m]))'Rule health is good and aligned evaluations agree, with expected cadence differences understood.