WIUNIX ACADEMY

Kubernetes Evicted Pods: distinguish node pressure, OOM and admission

Page 3 of 3. Review the earlier steps and prerequisites before continuing.

YOUR SYSTEM. YOUR STEPS.

Where are you working?

Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.

Ubuntu 26.10Reviewed against official documentation · 2026-11-05

USE YOUR OWN VALUES

Configure this tutorial

Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.

Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.

Set values for my environment3 editable values

LET’S BUILD

Optional: use current CPU/memory metrics within their limits

Use this branch only when the metrics API is present and authorized. The selected Pod must still exist and be running enough to have a current sample; for an evicted object, use the actual replacement’s identity deliberately or the existing historical monitoring instead. A missing metric is not zero usage.

kubectl top is a recent CPU/memory view from the metrics pipeline, not the kubelet’s complete eviction calculation. It does not supply disk bytes, inode/PID availability or the full filesystem layout, and a sample after recovery can miss the peak. Do not tune eviction thresholds from this one display.

Same authorized Bash shell on the administration host

date -u +%FT%TZ
k top node 'worker-a'
k top pod 'example-app' -n 'apps' --containers

How do I know it worked?

Current available metrics are labeled by sample/identity and kept separate from historical peaks and disk/inode/PID signals.

Ask for the signal that matches the pressure resource

For memory, the kubelet’s memory.available calculation is not interchangeable with an arbitrary free-memory number or a container limit. For disk, nodefs is the main node filesystem for work such as local volumes, disk-backed emptyDir and logs. Optional imagefs stores images and, without a separate containerfs, writable layers too. In a supported split layout, containerfs represents writable-layer storage. Compare byte and inode thresholds separately and confirm the runtime, version support and actual mounts: two or all three names can identify the same underlying filesystem, not distinct physical disks.

Container writable layers, logs and disk-backed emptyDir can contribute to local ephemeral storage. A memory-backed emptyDir consumes memory instead. PIDPressure is about available process identifiers, not CPU utilization. Ask the node owner for the effective version-specific kubelet thresholds, soft grace periods, reclaim behavior, reserved resources and matching historical measurements through the established node runbook.

Do not assume upstream defaults or infer all those settings from the administration host’s OS. No node-shell, proxy/config dump, cache deletion, runtime directory cleanup or filesystem repair command is supplied here. Deleting runtime files or images behind the runtime can damage active workloads and obscure the cause.

Know which disruption guarantees apply

Node-pressure eviction is a kubelet decision, distinct from the Eviction API used by administrative workflows. It does not respect a PodDisruptionBudget as protection from resource starvation. Hard thresholds can terminate immediately with a zero-second grace; soft-threshold handling is bounded by the configured eviction grace policy rather than a blanket promise to honor the Pod’s normal grace period.

The scheduler can avoid pressured nodes through conditions/taints, while kubelet admission can reject work after placement. A toleration permits consideration; it does not reserve memory or cure pressure. Scheduler priority/preemption is another mechanism from kubelet pressure selection. Do not remove taints, add broad tolerations, raise priorities or drain nodes simply to move a warning elsewhere.

Choose a reversible workload action with the actual owner

For sustained workload growth, the owner may reduce a restartable batch’s concurrency or fix resource requests/limits through its normal deployment source, with an explicit impact on throughput/backlog and a recorded previous setting. For node capacity or runtime/filesystem pressure, the node/storage owner chooses a supported capacity/reclamation plan using actual measurements. Neither action is a generic command derived from the Evicted label.

A critical writer needs its application consistency, quorum, fencing and restart/restore procedure before relocation or interruption. Coordinate replacement capacity first. Blind delete/drain/evict, force detach, clearing runtime directories, raising eviction thresholds or disabling protections can turn a recoverable pressure incident into data loss or a wider outage.

If an approved mitigation worsens errors or moves the incident to other nodes, restore the recorded configuration through its owning workflow and reconcile queued/retried work. Do not claim that rolling back a request or scheduler setting resurrects an already terminated Pod or reverses external application side effects.

Keep missing evidence separate from a negative finding

Each panel reads state only. Keep the original time, UID and failure message, and delegate an unavailable node/provider view instead of granting broad access or manufacturing a new failure.

Pod is Pending but not Evicted

Read scheduling conditions and the assigned node. Insufficient resource requests, affinity, storage binding and admission are different boundaries. Use the PVC guide for a storage scheduling failure rather than assuming memory pressure.

k get pod 'example-app' -n 'apps' -o jsonpath='{.spec.nodeName}{"\n"}{.status.phase}{"\t"}{.status.reason}{"\n"}{.status.conditions}{"\n"}'
k events -n 'apps' --for 'pod/example-app'

Pressure is false now

Keep the failure timestamp and current transition times. Eviction/reclamation may have relieved pressure before this sample. Obtain the incident interval from existing node monitoring; false now does not refute earlier pressure.

k get node 'worker-a' -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.lastTransitionTime}{"\t"}{.reason}{"\n"}{end}'

Metrics are missing or forbidden

Check permission and API discovery. A failed metrics request does not mean usage is zero and does not justify installing a metrics service during the incident.

k auth can-i get nodes.metrics.k8s.io
k api-resources --api-group=metrics.k8s.io

A replacement appears healthy

Compare UID, owner references and actual node. A new running Pod is useful recovery evidence, but the node pressure cause and application consistency still need verification.

k get pod 'example-app' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.ownerReferences}{"\n"}{.spec.nodeName}{"\n"}{.status.phase}{"\t"}{.status.reason}{"\n"}{.status.conditions}{"\n"}'
k get node 'worker-a' -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.lastTransitionTime}{"\n"}{end}'

LET’S BUILD

Verify the replacement and the resource trend together

After the owner’s separately approved mitigation, deliberately select the expected retained/replacement Pod and its actual node. Record the UID change and repeat conditions/events plus the application’s existing health/data checks. Confirm expected replica/quorum capacity through the workload owner; a green Pod alone is not full service recovery.

Observe comparable workload over the relevant period using existing monitoring. Pressure should stay within the owner’s accepted budget without repeated evictions, failed admission or new harm elsewhere. Less usage because all work stopped is not recovery. No configuration was changed by these diagnostic panels, and the retained failed object should be cleaned up only through the team’s normal evidence/retention policy.

Same authorized Bash shell on the administration host

k get pod 'example-app' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.ownerReferences}{"\n"}{.spec.nodeName}{"\n"}{.status.phase}{"\t"}{.status.reason}{"\n"}{.status.conditions}{"\n"}'
k get node 'worker-a' -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.lastTransitionTime}{"\n"}{end}'

How do I know it worked?

The intended workload and data checks pass under comparable work, and resource pressure/eviction recurrence is evaluated with time-correlated evidence.

GO TO THE SOURCE

Official references

Each environment states how it was reviewed. A documentation review does not claim a hands-on test of every version.

KEEP LEARNING

Your next step

Explore all tutorials ↗