WIUNIX ACADEMY

Build a Prometheus + Grafana monitoring stack

Page 4 of 5. Review the earlier steps and prerequisites before continuing.

YOUR SYSTEM. YOUR STEPS.

Where are you working?

Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.

Ubuntu 26.10Reviewed against official documentation · 2026-10-20

USE YOUR OWN VALUES

Configure this tutorial

Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.

Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.

Set values for my environment2 editable values

LET’S BUILD

Start Grafana and verify its health endpoint

Restart once after saving provisioning; enabling an already running service does not reload it. Require exact HTTP 200 and database ok, then inspect the listener: it must be 127.0.0.1:3000 before you sign in.

Monitoring server terminal.

(
set -euo pipefail
sudo chown root:grafana /etc/grafana/provisioning/datasources/prometheus.yaml
sudo chmod 0640 /etc/grafana/provisioning/datasources/prometheus.yaml
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl restart grafana-server
systemctl is-active grafana-server
ok=0
for attempt in $(seq 1 12); do
  if code=$(curl -sS --connect-timeout 2 --max-time 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:3000/api/health); then
    if [ "$code" = 200 ]; then ok=1; break; fi
  fi
  sleep 2
done
[ "$ok" = 1 ] || { echo 'Endpoint did not become ready: http://127.0.0.1:3000/api/health' >&2; exit 1; }
printf '%s -> 200\n' 'http://127.0.0.1:3000/api/health'
curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:3000/api/health | jq -e '.database == "ok"'
sudo ss -lntp 'sport = :3000'
)

EXAMPLE OUTPUT

Illustrative excerpt:
active
http://127.0.0.1:3000/api/health -> 200
true
Listener: 127.0.0.1:3000

How do I know it worked?

The grafana-server unit is active, the health endpoint prints an exact 200, and the JSON response reports database ok.

Grafana does not return HTTP 200

Inspect the Grafana service log and verify that the provisioning directory contains the expected data source file.

sudo journalctl -u grafana-server -n 100 --no-pager
sudo ls -l /etc/grafana/provisioning/datasources
sudo sed -n "1,120p" /etc/grafana/provisioning/datasources/prometheus.yaml

LET’S BUILD

Sign in to Grafana and confirm the data source

On your workstation, replace the two SSH placeholders with your existing login and server address. Keep the tunnel terminal open. Open http://127.0.0.1:13000 in your browser; the remote service remains private and the network leg is encrypted by SSH.

For a fresh installation, sign in with username admin and initial password admin, then immediately set a unique strong password. Do not reuse these defaults or put the new password in commands, source files or screenshots. The SSH server must permit local forwarding.

In Connections, find the provisioned Prometheus source. In Explore select it and run up. Both local instances must be 1; /api/health alone does not prove the data source works.

Workstation terminal, not the monitoring server. Replace YOUR_SSH_USER and MONITORING_SERVER before running.

ssh -N -o ExitOnForwardFailure=yes -L 127.0.0.1:13000:127.0.0.1:3000 YOUR_SSH_USER@MONITORING_SERVER

How do I know it worked?

You can sign in, change the initial password, and see the provisioned Prometheus data source available in Grafana.

LET’S BUILD

Create a first dashboard with practical PromQL panels

Create a new dashboard and add three panels that cover CPU usage, memory pressure and free filesystem space. The expressions below work well for a first validation and are easy to expand later.

Paste one expression per panel in the Grafana query editor. Adjust the panel titles to match the metric you choose.

Grafana dashboard panel editor in the browser.

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}

How do I know it worked?

Each panel shows live data instead of an empty or error state, proving that Grafana can query Prometheus successfully.

Optional: prepare the remote node safely

On the remote fresh amd64 node, first repeat the base package, node_exporter account, verified exporter download and loopback service steps above. Do not install Prometheus or Grafana there. This example uses monitoring IP 10.20.30.10 and remote IP 10.20.30.41; replace both consistently. On an already active UFW firewall with deny incoming, allow only the monitoring host. Inspect existing rules too: remove broad 9100 permits through your normal firewall management before exposing the exporter. Keep SSH allowed. If you use another firewall, implement the equivalent restriction before proceeding.

Monitoring server terminal.

(
set -euo pipefail
sudo ufw status verbose
sudo ufw status verbose | grep -q '^Status: active'
sudo ufw status verbose | grep -q '^Default: deny (incoming)'
sudo ufw allow proto tcp from 10.20.30.10 to 10.20.30.41 port 9100
sudo ufw status numbered
sudo install -d -m 0755 /etc/systemd/system/node_exporter.service.d
sudo nano /etc/systemd/system/node_exporter.service.d/private.conf
)

How do I know it worked?

The command completes successfully; stop and inspect any error before continuing.

Remote exporter private listener

On the remote node, save this override with its actual private IP. Clearing ExecStart is required before replacing it. The private network carries unauthenticated metrics: use a trusted network/VPN, or configure exporter TLS/auth separately.

File contents; save in the named file, not in the shell.

[Service]
ExecStart=
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.20.30.41:9100

How do I know it worked?

The override selects only the private address.

Start the private remote listener

Run on the remote node after reviewing its firewall. Then open the Prometheus configuration on the monitoring host using its earlier editor panel.

Monitoring server terminal.

(
set -euo pipefail
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl restart node_exporter
systemctl is-active node_exporter
sudo ss -lntp 'sport = :9100'
)

How do I know it worked?

The command completes successfully; stop and inspect any error before continuing.

Explore all tutorials ↗