WIUNIX ACADEMY
Kubernetes Evicted Pods: distinguish node pressure, OOM and admission
Page 2 of 3. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment3 editable values
LET’S BUILD
Read the failed Pod’s reason, message and identity
Record UID, creation time, owner references, assigned node, phase and reason before examining the events. Evicted plus a message that the node was low on a resource is useful pressure evidence. A different reason or merely a missing Pod does not establish node-pressure eviction.
Events expire and repeated events can be aggregated. Preserve the failure time and UID with existing central events/telemetry when available. If the object is already gone, ask the workload owner for the historical identity; do not recreate the incident or delete a replacement to obtain a cleaner display.
Same authorized Bash shell on the administration host
k get pod 'example-app' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.creationTimestamp}{"\n"}{.metadata.ownerReferences}{"\n"}{.spec.nodeName}{"\n"}{.status.phase}{"\t"}{.status.reason}{"\n"}{.status.message}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.reason}{"\t"}{.lastTransitionTime}{"\n"}{end}'
k events -n 'apps' --for 'pod/example-app'The specific failed Pod and its actual termination/scheduling boundary are identified from status and time-correlated evidence.
LET’S BUILD
Match the assigned node and read its pressure conditions
The guard compares the Pod’s assigned node with the form before inspecting that node. Stop this branch if scheduling never assigned one. Record the node UID, kubelet/runtime builds and condition transition times, not just the current Ready flag.
MemoryPressure, DiskPressure and PIDPressure refer to different node resources. A node can be Ready while pressure is present, or show pressure false after eviction already relieved it. Current conditions are a sample, not the complete incident history. A taint derived from a condition affects scheduling but is not itself a measurement of resource recovery.
Same authorized Bash shell on the administration host
observed_node=$(k get pod 'example-app' -n 'apps' -o jsonpath='{.spec.nodeName}') || exit 1
test -n "$observed_node" || { printf 'STOP: no assigned node; investigate scheduling first.\n' >&2; exit 1; }
test "$observed_node" = 'worker-a' || { printf 'STOP: actual node is %s; reconcile the public example first.\n' "$observed_node" >&2; exit 1; }
k get node 'worker-a' -o jsonpath='{.metadata.uid}{"\n"}{.status.nodeInfo.kubeletVersion}{"\n"}{.status.nodeInfo.containerRuntimeVersion}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.reason}{"\t"}{.lastTransitionTime}{"\t"}{.message}{"\n"}{end}{.spec.taints}{"\n"}{.spec.unschedulable}{"\n"}'The original Pod is tied to the correct node identity and time-scoped resource conditions; an unassigned or mismatched node stops the branch.
LET’S BUILD
Correlate node history with the Pod’s failure window
With separately authorized cluster-wide event access, inspect events for this exact node. Compare pressure transitions, reclamation/eviction messages and the Pod’s failure time. Conditions can oscillate and reporting is asynchronous; no single event or absence of an event is a complete chronology.
If the provider owns node management or event history has expired, hand the node UID, version, resource and time window to that owner. Do not treat a current green status as an explanation for an earlier failure.
Same authorized Bash shell on the administration host
k events --all-namespaces --for 'node/worker-a'Time-correlated node evidence supports the resource hypothesis, or the historical visibility gap is recorded.
LET’S BUILD
Separate a container OOM from Pod eviction
Read current and previous termination state for both init and ordinary containers. OOMKilled describes a container termination; a cgroup memory limit can cause it even when the whole node is not under pressure. Conversely, a node can run out of memory before the kubelet observes a threshold, and the kernel can choose a victim.
Use the Pod-level reason/message, node evidence and the operator’s kernel/kubelet telemetry to distinguish those paths. Exit 137 alone is not proof of OOM or eviction. An empty lastState is unavailable history, not a passed test; container restarts and a controller-created replacement Pod have different identities.
Same authorized Bash shell on the administration host
k get pod 'example-app' -n 'apps' -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\tinit current="}{.state}{"\tlast="}{.lastState}{"\trestarts="}{.restartCount}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{"\tcurrent="}{.state}{"\tlast="}{.lastState}{"\trestarts="}{.restartCount}{"\n"}{end}'Container termination/restart evidence is kept distinct from the Pod-level eviction decision and from a node-wide OOM.
LET’S BUILD
Read requests, limits and priority before blaming a QoS label
Inspect the actual Pod priority, QoS, per-container/init resources, any Pod-level resources/overhead and emptyDir settings. Requests are scheduling/allocation inputs; limits and current usage have different meanings. Init/sidecar/Pod-level accounting has specific rules, so do not simply add printed fields into a homemade node-available value.
For memory pressure, kubelet considers whether usage exceeds requests, Pod priority and usage relative to requests. Inodes and PIDs have no resource requests, so their starvation uses relative Pod priority for eviction order. Disk-pressure ranking also depends on the triggering signal and the nodefs/imagefs/containerfs layout. QoS can help interpret some memory cases but is not the direct general eviction-order key or protection for disk/ephemeral-storage use. High priority and Guaranteed QoS do not make a Pod immune; raising priority shifts consequences to other workloads without adding capacity.
Same authorized Bash shell on the administration host
k get pod 'example-app' -n 'apps' -o jsonpath='{.spec.priorityClassName}{"\tpriority="}{.spec.priority}{"\n"}{.status.qosClass}{"\n"}{.spec.overhead}{"\n"}{.spec.resources}{"\n"}{range .spec.initContainers[*]}{.name}{"\tinit resources="}{.resources}{"\n"}{end}{range .spec.containers[*]}{.name}{"\tresources="}{.resources}{"\n"}{end}{range .spec.volumes[*]}{.name}{"\temptyDir="}{.emptyDir}{"\n"}{end}'The affected workload’s real resource/priority contract is available without modifying it or treating QoS as an immunity guarantee.
LET’S BUILD
Distinguish node allocatable capacity from current availability
This optional cluster-wide Pod list shows the occupants of the affected node and their current reasons/priorities. Node capacity is not the same as allocatable; reservations and eviction policy consume part of the budget. Allocatable also is not a live measurement of bytes of free memory or disk.
Compare the scheduler’s request accounting and existing time-series usage with the node owner. A scheduler FailedScheduling message about insufficient requests/capacity is different from a kubelet refusing a newly assigned Pod during admission or evicting a running Pod. Read which component emitted the message and when; do not fix all three by deleting Pods or removing pressure taints.
Same authorized Bash shell on the administration host
k get node 'worker-a' -o jsonpath='{.status.capacity}{"\n"}{.status.allocatable}{"\n"}'
k get pods --all-namespaces --field-selector='spec.nodeName=worker-a' --chunk-size=100 -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,REASON:.status.reason,PRIORITY:.spec.priority,QOS:.status.qosClass'Node budget and workload scope are understood without confusing reported capacity, requested allocation and live eviction signals.