Linux I/O latency: connect iostat, PSI and service pressure

Use vmstat for the same incident period. Rate and CPU columns in its first data row are averages since boot; read the following rows for interval comparisons. Process and memory fields are instantaneous in every row. Look at blocked tasks, swap-in/swap-out and CPU fields together. The wa column is CPU idle time associated with outstanding I/O, not the percentage of application time lost to I/O.

A workload may stall on I/O while other tasks keep CPUs busy, leaving wa small. Memory reclaim and swapping can create storage traffic without a new database read surge. These signals support a hypothesis; they do not identify an individual disk or authorize changing swap, cache or kernel writeback settings.

date -u +%FT%TZ
LC_ALL=C vmstat 1 6

pidstat samples task-accounted reads and writes. This panel uses process names rather than full command-line arguments, which may contain private values. A process with high write accounting is a candidate owner, not proof that every device request completed synchronously for that PID: buffered writeback and shared filesystems complicate attribution.

Zero or absent iodelay is inconclusive when task delay accounting is unavailable or disabled. The sysstat FAQ documents the kernel accounting requirements. Do not enable delay accounting or broaden procfs access as an incident shortcut. Resolve visibility with the host owner and keep the documented blind spot in the incident record.

date -u +%FT%TZ
LC_ALL=C pidstat -d -p ALL 1 5

Set example to the existing systemd service basename, without the .service suffix. Inspect its identity and ControlGroup before using a cgroup path. This optional branch requires cgroup v2 and host-level access; it does not create or move cgroups. A container’s private cgroup namespace may not expose the host path.

Combine the actual cgroup2 mount point with the returned ControlGroup and put that existing absolute directory into /sys/fs/cgroup/system.slice/example.service. The initial example is intentionally not a discovered service. A missing unit or empty ControlGroup means stop this branch, not copy a guessed directory.

systemctl show 'example.service' -p Id -p LoadState -p ActiveState -p MainPID -p ControlGroup
findmnt -t cgroup2 -o TARGET,FSTYPE,OPTIONS

Read the discovered service directory twice. io.stat counts bytes and operations by major:minor; compare each device’s delta over the same interval and map it back to lsblk. io.max can reveal a configured ceiling. A max entry does not prove that the service reached that ceiling, and absence of a local cap does not exclude a limit at a parent cgroup or storage provider.

Service-level pressure can be high while the host looks comfortable. Inspect the relevant ancestor policy with the service owner if needed; do not change io.max, io.weight or controller delegation to make numbers improve. Some kernels or cgroup arrangements do not expose every file; missing interfaces are not zeros.

(
set -eu
cg='/sys/fs/cgroup/system.slice/example.service'
test -d "$cg"
test "$(findmnt -n -o FSTYPE --target "$cg")" = cgroup2
for sample in 1 2; do
  date -u +%FT%TZ
  for name in io.pressure io.stat io.max memory.pressure; do
    printf '\n%s\n' "$name"
    if test -r "$cg/$name"; then cat "$cg/$name"; else printf 'unavailable or unreadable\n' >&2; fi
  done
  if test "$sample" = 1; then sleep 5; fi
done
)