Wiunix
Talk to an Engineer
← Start of tutorial

WIUNIX ACADEMY

Build a Prometheus + Grafana monitoring stack

Page 5 of 5. Review the earlier steps and prerequisites before continuing.

YOUR SYSTEM. YOUR STEPS.

Where are you working?

Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.

Ubuntu 26.10Reviewed against official documentation ·

USE YOUR OWN VALUES

Configure this tutorial

Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.

Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.

Set values for my environment2 editable values
16

LET’S BUILD

Add another Linux node to the same stack

On the monitoring host, retain both local targets and append the remote private endpoint as shown. If you already added other jobs, merge this target into the existing file instead of replacing those jobs.

YAML · /etc/prometheus/prometheus.yml

File contents; save in the named file, not in the shell.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - 127.0.0.1:9090

  - job_name: node-exporter
    static_configs:
      - targets:
          - 127.0.0.1:9100
          - 10.20.30.41:9100
How do I know it worked?

The saved configuration contains the two local targets and the intended remote address; the next panel validates and restarts it.

The remote target stays down

From the monitoring host inspect reachability and lastError. On the remote host rerun its listener and firewall checks above; a local listener on the monitoring host says nothing about the remote bind.

Bash
# Monitoring host: transport and target diagnostics
curl -fsS --connect-timeout 2 --max-time 5 http://10.20.30.41:9100/metrics
curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:9090/api/v1/targets | jq .

Validate, restart and prove the remote scrape

Run only on the monitoring host. A restart briefly interrupts scraping; configuration validation must succeed first. Then require the exact remote instance to report up=1.

Bash

Monitoring server terminal.

(
set -euo pipefail
sudo -u prometheus /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus
ok=0
for attempt in $(seq 1 12); do
  if code=$(curl -sS --connect-timeout 2 --max-time 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9090/-/ready); then
    if [ "$code" = 200 ]; then ok=1; break; fi
  fi
  sleep 2
done
[ "$ok" = 1 ] || { echo 'Endpoint did not become ready: http://127.0.0.1:9090/-/ready' >&2; exit 1; }
printf '%s -> 200\n' 'http://127.0.0.1:9090/-/ready'
response=$(mktemp)
trap 'rm -f "$response"' EXIT
ok=0
for attempt in $(seq 1 12); do
  if code=$(curl -sS --connect-timeout 2 --max-time 5 -G \
    --data-urlencode 'query=up{job="node-exporter",instance="10.20.30.41:9100"}' -o "$response" -w '%{http_code}' \
    http://127.0.0.1:9090/api/v1/query); then
    if [ "$code" = 200 ] && jq -e '.status == "success" and .data.resultType == "vector" and (.data.result | length == 1) and .data.result[0].value[1] == "1"' "$response" >/dev/null; then
      ok=1; break
    fi
  fi
  sleep 5
done
[ "$ok" = 1 ] || { cat "$response" >&2; echo 'Remote target must have up=1' >&2; exit 1; }
jq . "$response"
)
How do I know it worked?

The command completes successfully; stop and inspect any error before continuing.

17

LET’S BUILD

Review versions and routine maintenance points

Routine maintenance for this stack means tracking new Prometheus and node_exporter releases, keeping the Grafana package current, and watching disk growth under the Prometheus data directory. Prometheus and node_exporter are not updated automatically because they were installed from tarballs.

Run the checks below during maintenance windows so you can see the currently installed versions and active service state before you plan an upgrade.

Bash

Monitoring server terminal.

(
set -euo pipefail
prometheus --version
node_exporter --version
dpkg -l grafana | awk '/^ii/ {print $2, $3}'
systemctl is-active prometheus node_exporter grafana-server
sudo du -sh /var/lib/prometheus
df -h /var/lib/prometheus
)
How do I know it worked?

You can read the installed component versions, confirm that all services are active, and see the current size of the Prometheus data directory.

18

LET’S BUILD

Stop safely and verify a consistent backup

Optional retirement interrupts dashboards and scraping. Stop services before archiving to keep Prometheus TSDB and Grafana SQLite consistent. The archive preserves full relative paths, so configuration and data never collide. A failed stop, archive, checksum or read-back aborts this block. This is retirement, not an automatic rollback to an older binary.

All original configuration and data remain in place. Inspect the printed backup and copy it to your existing protected backup storage before any further removal. A local checksum and archive comparison establish local integrity, not disaster recovery. Keep this backup private.

Bash

Monitoring server terminal.

(
set -euo pipefail
sudo systemctl disable --now grafana-server prometheus node_exporter
sudo sh -eu <<'BACKUP'
umask 077
install -d -m 0700 /root/academy-backups
backup=$(mktemp -d /root/academy-backups/monitoring-XXXXXXXX)
tar -C / -cpf "$backup/monitoring.tar" \
  etc/prometheus var/lib/prometheus etc/grafana var/lib/grafana \
  etc/systemd/system/prometheus.service etc/systemd/system/node_exporter.service \
  etc/systemd/system/grafana-server.service.d/academy-local.conf \
  etc/apt/sources.list.d/grafana.list etc/apt/keyrings/grafana.asc
cd "$backup"
sha256sum monitoring.tar > SHA256SUMS
sha256sum -c SHA256SUMS
tar -tf monitoring.tar >/dev/null
tar -C / -df monitoring.tar
printf 'Verified local archive: %s/monitoring.tar\n' "$backup"
BACKUP
)
How do I know it worked?

Services are stopped, the archive passes checksum and comparison, and original data/configuration are retained. If anything fails, keep the originals and investigate.

Optional: remove installed programs, retain data

Only after verifying and independently retaining the backup above, remove the three tutorial-installed binaries and the two tutorial-created units. The Grafana remove operation is deliberately not a purge; do not use autoremove. Data, configuration, system accounts, repository key and the loopback override remain available for recovery. Do not run this on a host with another installation owning these paths.

Bash

Monitoring server terminal.

(
set -euo pipefail
for unit in grafana-server prometheus node_exporter; do
  state=$(systemctl show -p ActiveState --value "$unit")
  case "$state" in inactive|failed) ;; *) echo "Stop $unit before removal" >&2; exit 1 ;; esac
done
sudo apt-get remove grafana
sudo rm -f /etc/systemd/system/prometheus.service /etc/systemd/system/node_exporter.service
sudo rm -f /usr/local/bin/prometheus /usr/local/bin/promtool /usr/local/bin/node_exporter
sudo systemctl daemon-reload
)
How do I know it worked?

The command completes successfully; stop and inspect any error before continuing.

Recover retained state and continue

For a simple stop, the original data and configuration still exist: reinstall the same recorded component versions if removed, validate Prometheus configuration and repeat the start/readiness/data-source checks above. For recovery from the archive, first verify its checksum and restore into an isolated directory; review ownership and paths before restoring stopped services. Do not extract a backup over running databases or downgrade a TSDB without upstream compatibility guidance.

Next, choose retention and capacity limits for your workload and add alert rules with tested notification delivery. A dashboard alone does not page an operator during an outage.

How do I know it worked?

Recovery uses retained state and a reviewed compatible version; alerting remains an explicit next task.

GO TO THE SOURCE

Official references

Each environment states how it was reviewed. A documentation review does not claim a hands-on test of every version.

Explore all tutorials ↗