WIUNIX ACADEMY
Prometheus and Grafana with Docker Compose: private monitoring and recovery
Page 4 of 6. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment11 editable values
LET’S BUILD
Validate before starting any service
Normalize permissions on non-secret files only, validate Compose and use the exact pinned image’s promtool. Pulling images is not starting the stack. Inspect effective configuration for loopback bindings, digest pins and read-only mounts; it contains the secret file path but not its contents. JSON syntax checks do not establish that every Grafana option works; the live dashboard check follows.
Dedicated Linux host; Bash shell
sudo find '/srv/academy-monitor/config' '/srv/academy-monitor/dashboards' -type f -exec chmod 0644 {} +
sudo docker compose -f '/srv/academy-monitor/compose.yaml' config --quiet
sudo docker compose -f '/srv/academy-monitor/compose.yaml' config
sudo python3 -m json.tool '/srv/academy-monitor/config/targets/services.json' >/dev/null
sudo python3 -m json.tool '/srv/academy-monitor/dashboards/compose.json' >/dev/null
sudo docker compose -f '/srv/academy-monitor/compose.yaml' pull
sudo docker compose -f '/srv/academy-monitor/compose.yaml' run --rm --no-deps --entrypoint /bin/promtool prometheus check config /etc/prometheus/prometheus.yml
All validation commands succeed; promtool reports SUCCESS before launch.
LET’S BUILD
Start, then prove readiness
Start only this named project. The Compose ps state alone is insufficient: retry the probes for up to a minute during initial database creation. Prometheus healthy indicates the process responds; ready means it can serve queries. Grafana health must report database ok. No public listener should appear. A healthy monitoring stack still cannot alert you if the entire host disappears.
Dedicated Linux host; Bash shell
sudo docker compose -f '/srv/academy-monitor/compose.yaml' up -d
sudo docker compose -f '/srv/academy-monitor/compose.yaml' ps
curl --fail --show-error http://127.0.0.1:19090/-/healthy
curl --fail --show-error http://127.0.0.1:19090/-/ready
curl --fail --show-error http://127.0.0.1:13000/api/health
sudo docker compose -f '/srv/academy-monitor/compose.yaml' port prometheus 9090
sudo docker compose -f '/srv/academy-monitor/compose.yaml' port grafana 3000
sudo ss -lntp
Both APIs ready; published host addresses are only 127.0.0.1 and data directories remain writable by intended users.
LET’S BUILD
Open a local-only SSH tunnel
On your workstation, use the existing authorized SSH account and verified host key. Keep the session open. If either local port is occupied, select free ports above before copying so tunnel and browser URLs stay synchronized. Host-key mismatch is a stop condition, not permission to disable verification. This is encrypted SSH transport; the HTTP URLs below never cross the internet directly.
Your authorized workstation; OpenSSH
ssh -N -o ExitOnForwardFailure=yes -p 22 -L 127.0.0.1:13000:127.0.0.1:13000 -L 127.0.0.1:19090:127.0.0.1:19090 operator@192.0.2.40
Tunnel remains connected without forwarding failures.
Inspect the provisioned dashboard
In the workstation browser open http://127.0.0.1:13000/ and sign in as admin with the protected bootstrap password. Open Dashboards → Academy → Compose service health. Both up series should become 1 after two scrape intervals. Open http://127.0.0.1:19090/targets to confirm the actual endpoint names and scrape errors. Create separate least-privilege users for routine viewing through your normal identity process; do not share the bootstrap admin. Changing the existing admin password requires Grafana’s account flow or documented CLI recovery, not editing the bootstrap file. For public multi-user HTTPS/SSO deployment, separately design proxy trust, root URL, cookies, CSRF, TLS and egress; do not simply publish port3000.
LET’S BUILD
Verify collection through the query API
Run on the host after at least 30 seconds. Both expected targets must be present and healthy; a success HTTP status with an empty vector is not a pass. No password goes into these private Prometheus queries. Keep operational target output out of public logs.
Dedicated Linux host; Bash shell
curl --fail --show-error -G http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=up'
curl --fail --show-error http://127.0.0.1:19090/api/v1/targets
sudo docker stats --no-stream
sudo du -sh '/srv/academy-monitor/prometheus-data' '/srv/academy-monitor/grafana-data'
Two expected up=1 series; inspect actual container resources and disk use.
LET’S BUILD
Observe a real scrape failure
Optional lab-only exercise: stopping Grafana interrupts its UI. Leave Prometheus running and use its query API after 45 seconds. Grafana up must become 0 while Prometheus stays 1; then start Grafana and confirm both recover. This proves a local scrape failure, not alert delivery or high availability. After recovery, the dashboard history should show the outage. An external dead-man check and separately operated alert receiver are needed to detect whole-host loss.
Dedicated Linux host; Bash shell
sudo docker compose -f '/srv/academy-monitor/compose.yaml' stop grafana
sleep 45
curl --fail --show-error -G http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=up'
sudo docker compose -f '/srv/academy-monitor/compose.yaml' start grafana
sleep 30
curl --fail --show-error -G http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=up'
curl --fail --show-error -G http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=min_over_time(up{service="grafana"}[10m])'
Grafana transitions 1→0→1; the ten-minute minimum retains the failure.