Linux I/O latency: connect iostat, PSI and service pressure

Use an authorized host shell, not an arbitrary container shell: mount, PID and cgroup namespaces can show different scopes. Compare the selected distribution with the actual release and kernel. sysstat supplies iostat and pidstat; procps supplies vmstat on DEB systems, while RPM and Arch families call that package procps-ng.

The checks below install nothing. Resolve missing tools through the host’s approved package-management process before continuing. The following optional installation panel changes packages; it is not a required step on an incident host. No sar collector, timer or service needs to be enabled for these short interactive samples.

Use this Ubuntu release’s official repositories. The 26.04/24.04 paths are LTS; 26.10 is the interim release with a shorter support window.

cat /etc/os-release
uname -r
ps -p 1 -o comm=
dpkg-query -W sysstat procps util-linux coreutils systemd
command -v iostat pidstat vmstat findmnt lsblk systemctl date
iostat -V
pidstat -V

Skip this panel when the required tools already exist. On an approved lab or maintenance window, review the package transaction and obtain tools only from the selected distribution’s official enabled repositories. Package installation can start package-owned units; review the transaction and resulting unit state rather than enabling historical collection just for this lesson.

The remaining observations are read-only, but still consume resources. Keep sample counts bounded and evidence private. Existing AppArmor, SELinux, procfs restrictions and delegation boundaries remain in place; a permission error is a visibility limit to resolve with the host owner, not a reason to disable protection.

sudo apt-get update
sudo apt-get install sysstat procps util-linux coreutils
command -v iostat pidstat vmstat findmnt lsblk

Set /var/lib to an existing application data directory. findmnt identifies its filesystem in this mount namespace. lsblk then exposes the block-device tree and major:minor identifiers. Follow a mapper, LVM or RAID device to its backing devices rather than assuming the first nvme name belongs to this application.

Keep each layer distinct. A request may appear at both a logical volume and its physical device; adding their byte counters double-counts the same work. If the source is a network filesystem or an overlay, stop the local-device attribution branch and investigate the real backing path with its owner.

test -d '/var/lib' || exit 1
findmnt --target '/var/lib' -o TARGET,SOURCE,FSTYPE,OPTIONS
lsblk -o NAME,KNAME,MAJ:MIN,TYPE,SIZE,FSTYPE,MOUNTPOINTS

PSI measures task stalls rather than device busy time. For I/O, some means at least one task is stalled; full means all non-idle tasks in that scope are stalled together. avg10, avg60 and avg300 are percentage trends for those windows, while total is cumulative stall time in microseconds. full is a subset of some, so never add them.

Read host I/O and memory pressure twice, with UTC timestamps. A short spike can increase total while barely moving the longer average. Rising memory pressure and swap activity can accompany I/O stalls; that correlation needs the next observations before concluding the device itself is slow. A missing pressure file means PSI is unavailable in this kernel or scope, not that contention is absent.

date -u +%FT%TZ
for resource in io memory; do
  printf '\n%s pressure\n' "$resource"
  if test -r "/proc/pressure/$resource"; then
    cat "/proc/pressure/$resource"
  else
    printf 'PSI unavailable or unreadable in this scope\n' >&2
  fi
done
sleep 5
date -u +%FT%TZ
for resource in io memory; do
  if test -r "/proc/pressure/$resource"; then cat "/proc/pressure/$resource"; fi
done

Run during the observed slowdown and compare with a similar normal workload. The -y option skips iostat’s since-boot report. Read the relevant device layers from the previous map: r_await and w_await include time queued and serviced, aqu-sz describes average outstanding requests, and MiB/s and operations/s show the workload shape.

A high %util means requests were active for much of the interval. It is not a universal capacity meter for parallel NVMe, RAID or virtual devices. Low throughput with increasing await can reflect small synchronous requests, a throttled backend or errors; high throughput with stable application latency may be useful work. Do not calculate service time by subtracting these averages.

date -u +%FT%TZ
LC_ALL=C iostat -d -x -m -y 1 5