Atomic nftables changes: rehearse timeout rollback before risking remote access

Run this whole panel together. A root shell takes an exclusive lock, rechecks the candidate, creates the detached timeout timer and verifies it is active before loading the candidate. The timeout service takes the same lock, so if its deadline arrives during a slow apply, it waits and restores after that transaction releases the lock. This avoids restoring first and then accidentally applying a late candidate.

The lock coordinates only this fixture’s two commands. It does not lock out firewalld, another operator or a process that ignores the lock. Do not modify the snapshot, namespace names or files while the timer is armed. Loss of the terminal does not normally cancel the system timer, but reboot, manager failure, namespace deletion or an unsuccessful restore still defeats recovery.

The immediate ping is expected to receive no echo reply. Exit 1 alone can reflect loss on another boundary, so confirm the intended drop counter increased before attributing the loss to this rule. A different ping error stops the panel. Leave rollback armed. If the ping succeeds, the timer may already have restored the baseline or the candidate did not affect the tested path; inspect the actual result.

ip_bin=$(sudo sh -c 'command -v ip') || exit 1
nft_bin=$(sudo sh -c 'command -v nft') || exit 1
flock_bin=$(sudo sh -c 'command -v flock') || exit 1
sudo /bin/sh -c 'set -eu
exec 9>"$1/change.lock"
"$4" -x 9
test "$(systemctl show -p LoadState --value "$2.timer")" = not-found
test "$(systemctl show -p LoadState --value "$2.service")" = not-found
"$5" netns exec "$6" "$7" -c -f "$1/candidate.nft"
systemd-run --unit="$2" --on-active="$3"s --timer-property=AccuracySec=1s --property=Type=oneshot --property=RemainAfterExit=yes "$4" -x "$1/change.lock" "$5" netns exec "$6" "$7" -f "$1/rollback.nft"
systemctl is-active --quiet "$2.timer"
"$5" netns exec "$6" "$7" -f "$1/candidate.nft"
' sh '/var/lib/academy-nft-rollback-lab' 'academy-nft-rollback' '90' "$flock_bin" "$ip_bin" 'academy-nft-server' "$nft_bin" || exit 1
if sudo ip netns exec 'academy-nft-client' ping -c 1 -W 2 '192.0.2.1'; then
  printf 'UNEXPECTED: candidate did not block the lab probe\n' >&2
  exit 1
else
  probe_status=$?
  if test "$probe_status" -ne 1; then
    printf 'Probe error, not evidence of a firewall drop: exit=%s\n' "$probe_status" >&2
    exit "$probe_status"
  fi
  printf 'No echo reply; inspect the intended drop counter; leave rollback armed\n'
fi
sudo ip netns exec 'academy-nft-server' nft -n list table inet '"academy_guard"' || exit 1
systemctl list-timers --all 'academy-nft-rollback.timer'

Wait through the configured interval, then inspect the restore service, table and fresh client probe. The relative timer is not real-time scheduling; allow for manager load and the lock if a result is not yet available. A timer firing is not proof its command succeeded.

Accept recovery only when the service result is successful, the baseline allow is back, and the new probe succeeds. This is a lab ICMP recovery result, not proof that an existing production SSH session or a new SSH login would survive a real host-policy change.

sleep '90'
sudo systemctl show 'academy-nft-rollback.service' -p ActiveState -p Result -p ExecMainStatus
sudo journalctl -u 'academy-nft-rollback.service' --no-pager -n 40
test "$(systemctl show -p ActiveState --value 'academy-nft-rollback.service')" = active || exit 1
test "$(systemctl show -p SubState --value 'academy-nft-rollback.service')" = exited || exit 1
test "$(systemctl show -p Result --value 'academy-nft-rollback.service')" = success || exit 1
test "$(systemctl show -p ExecMainStatus --value 'academy-nft-rollback.service')" = 0 || exit 1
sudo ip netns exec 'academy-nft-server' nft -n list table inet '"academy_guard"'
sudo ip netns exec 'academy-nft-client' ping -c 2 -W 2 '192.0.2.1' || exit 1

Use the VM host console to inspect the failure first. The host is outside the filtered namespace. The following restore takes the same lock and reloads only the captured table, then checks the probe. It preserves the files and timer evidence for diagnosis.

If the namespace or table was deleted or another object now owns the name, stop rather than making the snapshot fit an unknown state. A restore that cannot delete its expected table fails; it is not a reason to flush the host or disable the firewall.

ip_bin=$(sudo sh -c 'command -v ip') || exit 1
nft_bin=$(sudo sh -c 'command -v nft') || exit 1
sudo flock -x '/var/lib/academy-nft-rollback-lab/change.lock' "$ip_bin" netns exec 'academy-nft-server' "$nft_bin" -f '/var/lib/academy-nft-rollback-lab/rollback.nft' || exit 1
sudo ip netns exec 'academy-nft-client' ping -c 2 -W 2 '192.0.2.1' || exit 1

Runtime nft changes vanish when their namespace is destroyed. This experiment writes no boot policy. Inspect the installed service file before adapting any real persistent policy: Ubuntu/Debian and Arch normally use their nftables main configuration file, while RHEL-family service loading uses its sysconfig include file. Environment notes identify the native paths; the actual unit is the final authority.

Reload/start/stop behavior can run broad commands. For example, the reviewed Debian nftables unit flushes the entire host ruleset when stopped. Do not stop or restart that service to clean up a namespace table. With firewalld, runtime and permanent state are separate; with NetworkManager, connection zone settings may be reapplied. Persist a separately approved change only through its owning manager, then qualify its reload/reboot recovery in a disposable clone first.

systemctl cat nftables.service
systemctl show nftables.service -p FragmentPath -p DropInPaths -p ExecStart -p ExecReload -p ExecStop
systemctl show firewalld.service ufw.service NetworkManager.service -p Id -p LoadState -p ActiveState