WIUNIX ACADEMY
Prometheus and Grafana with Docker Compose: private monitoring and recovery
Page 5 of 6. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment11 editable values
LET’S BUILD
Replace targets atomically
Update only the public metadata label in the target file. File SD reloads this directory without restarting Prometheus. The new label creates a new time series; old labels remain in historical data until retention expires. Arbitrary user IDs or request URLs must never become unbounded labels. For the main Prometheus YAML, validate then recreate the service; an editor’s atomic replacement of a bound file can leave the old inode in the running container. No web lifecycle/admin API is enabled.
Dedicated Linux host; Bash shell
sudo tee '/srv/academy-monitor/config/targets/services.json.new' >/dev/null <<'JSON'
[{"targets":["grafana:3000"],"labels":{"service":"grafana","stack":"academy-monitor","stage":"lab"}}]
JSON
sudo python3 -m json.tool '/srv/academy-monitor/config/targets/services.json.new' >/dev/null
sudo chmod 0644 '/srv/academy-monitor/config/targets/services.json.new'
sudo mv '/srv/academy-monitor/config/targets/services.json.new' '/srv/academy-monitor/config/targets/services.json'
sleep 45
curl --fail --show-error -G http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=up{stage="lab"}'
New stage=lab series is present and up=1 without Prometheus restart.
LET’S BUILD
Diagnose the actual failing layer
For permission denied, compare mounts and numeric owners; do not recursively chown an unrelated path. A directory where a file is expected means a missing bind source. For target DNS errors, compare network membership and service names; host curl success does not prove container reachability. For login failure after changing the secret, remember initialization happens once. OOMKilled or restart loops require a series/memory review, not endless restarts. Read logs locally because real exporters may include sensitive labels. Do not dump process environment or secret contents.
Dedicated Linux host; Bash shell
sudo docker compose -f '/srv/academy-monitor/compose.yaml' ps -a
sudo docker compose -f '/srv/academy-monitor/compose.yaml' logs --tail=80 prometheus grafana
sudo docker inspect --format '{{.Name}} user={{.Config.User}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' $(sudo docker compose -f '/srv/academy-monitor/compose.yaml' ps -aq)
sudo stat -c '%u:%g %a %n' '/srv/academy-monitor/prometheus-data' '/srv/academy-monitor/grafana-data' '/srv/academy-monitor/secrets/grafana-admin'
sudo docker network inspect 'academy-monitor_monitor'
df -h '/srv/academy-monitor'
A concrete mount, ownership, DNS, authentication or resource cause is identified before changes.
Treat retention as a budget, not a backup
Prometheus removes old blocks when either the time or size limit is reached first. Its GB suffix represents binary gigabytes; the size flag covers blocks, not every WAL/head byte, and cleanup is not instantaneous. Leave roughly 15–20% extra headroom beyond the block budget, plus separate allowance for everything else; size actual ingest and compaction peaks. Resource limits can OOM-kill a growing TSDB. Watch disk, cardinality, scrape duration, failures and container restarts. The single local disk and host are failure domains; retention, restart and a second container do not create HA or an off-host backup.
LET’S BUILD
Take a stopped, consistent backup
This causes a monitoring/UI gap. Stop both services cleanly before copying the TSDB and Grafana SQLite database. The archive contains the bootstrap secret, database credentials/sessions, configuration and image pins: keep root-only, encrypt it with your approved backup system and transfer off-host without printing secrets. The trap resumes the original stack even if copying fails. Do not back up live SQLite with ordinary file copies or copy only Prometheus block directories and call it a complete backup. The fixed archive name refuses overwrite; move an accepted prior backup through your normal retention process before repeating.
Dedicated Linux host; Bash shell
sudo bash -eu <<'BACKUP'
archive='/srv/academy-monitor-backups/compose-stack.tar'
[[ ! -e "$archive" ]]
docker compose -f '/srv/academy-monitor/compose.yaml' stop -t 60
trap 'docker compose -f "/srv/academy-monitor/compose.yaml" start' EXIT
[[ -z "$(docker compose -f '/srv/academy-monitor/compose.yaml' ps --status running -q)" ]]
umask 077
tar --numeric-owner -cpf "$archive" -C '/srv/academy-monitor' .
cd '/srv/academy-monitor-backups'
sha256sum compose-stack.tar > compose-stack.tar.sha256
sha256sum -c compose-stack.tar.sha256
BACKUP
Archive and checksum exist with private permissions; original services resume and readiness probes pass again.
LET’S BUILD
Rehearse same-host recovery without erasing data
Use only your own trusted archive. Verify its checksum, then stop/remove only this project’s containers/network; no volumes or images are deleted. Rename the entire existing stack aside and extract into a new empty directory at the same original path, preserving numeric owners. Absolute bind paths mean moving the restored tree elsewhere is not enough. The previous tree remains your rollback. This is disruptive and needs enough disk for both trees. On a replacement host, install supported Docker, restore the original paths/users and secrets, review subnet/ports, and pull the exact digests before starting; offline recovery also needs an approved image archive. Never run old and restored copies with the same project/network concurrently.
Dedicated Linux host; Bash shell
sudo bash -eu <<'RESTORE'
cd '/srv/academy-monitor-backups'
sha256sum -c compose-stack.tar.sha256
[[ ! -e '/srv/academy-monitor.before-restore' ]]
docker compose -f '/srv/academy-monitor/compose.yaml' down --timeout 60
mv '/srv/academy-monitor' '/srv/academy-monitor.before-restore'
install -d -m 0755 '/srv/academy-monitor'
tar --numeric-owner -xpf '/srv/academy-monitor-backups/compose-stack.tar' -C '/srv/academy-monitor'
docker compose -f '/srv/academy-monitor/compose.yaml' config --quiet
docker compose -f '/srv/academy-monitor/compose.yaml' up -d
RESTORE
Original absolute paths and ownership restored; keep the previous directory until acceptance.
Prove restored history and identity
Repeat readiness/target probes from part3. In the browser, use the pre-backup account and confirm the provisioned dashboard plus historical data from before the backup. Verify users/preferences and any real data-source secrets without exposing them. Compare archive-to-tree bytes only while the restored services are stopped; running databases legitimately change. Recovery time and acceptable data loss must be measured in your environment. A checksum alone proves file integrity, not successful restore. If acceptance fails, use the next rollback panel; keep the failed restore for diagnosis.