WIUNIX ACADEMY
Kubernetes PVC Pending and CSI: diagnose binding, attach and mount
Page 3 of 3. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment9 editable values
LET’S BUILD
Separate attachment from node-side mounting
This optional query needs cluster-wide VolumeAttachment list permission. It reads the list in API chunks and prints only entries for the chosen PV; it is not a namespace-scoped authorization. Match the attacher and node to the Pod’s placement, then read attached and any attach/detach error. There may be no such object for a driver that skips attach.
An attached=true result still does not prove NodeStage/NodePublish mounting, permissions or application access. FailedAttachVolume, Multi-Attach and FailedMount event reasons point to different stages; read the full timestamped message. For a single-writer backend, verify the old writer’s actual stop/fencing with its owner before any move. Deleting the attachment or forcing detach can create two writers and corrupt data.
Same authorized Bash shell on the administration host
k get volumeattachments.storage.k8s.io --chunk-size=100 -o jsonpath='{range .items[?(@.spec.source.persistentVolumeName=="example-volume")]}{.metadata.name}{"\t"}{.spec.attacher}{"\t"}{.spec.nodeName}{"\tattached="}{.status.attached}{"\n"}{.status.attachError.message}{"\n"}{.status.detachError.message}{"\n"}{end}'
k events -n 'apps' --for 'pod/example-app'Any applicable attachment is matched to this PV/node and distinguished from a mount or application failure.
LET’S BUILD
Optional: read the identified CSI component’s bounded logs
Ask the storage operator which installed controller/node Pod and container correspond to this driver and failure time. Enter those exact discovered names, inspect the container names/images first, then request only the relevant recent tail. Controller provisioning errors belong with the provisioner; attachment errors with the attacher; node-stage/publish errors with the node plugin and kubelet.
Driver-specific backend errors may disclose internal endpoints or identifiers; keep this output private. A permission failure or a provider-managed component you cannot inspect is a visibility limit for its owner, not a reason to run privileged debug Pods, fetch Secrets or restart every CSI component. Check the installed driver/sidecar compatibility matrix before interpreting a missing feature.
Same authorized Bash shell on the administration host
k get pod 'example-csi-controller' -n 'kube-system' -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
k logs 'example-csi-controller' -n 'kube-system' -c 'csi-provisioner' --since=10m --tail=80 --timestampsA specific component and bounded failure window are identified; unseen managed components are explicitly handed to their owner.
Choose the owner and recovery path for the failed stage
For no eligible consumer node, fix the agreed workload placement/capacity issue through the normal platform source. For dynamic provisioning failure, the storage owner checks class parameters, credentials by their protected method, backend quota/capacity and controller compatibility. For an existing PV, reconcile its exact claim UID and topology rather than changing names to make the status match.
For attach/mount failure, keep the volume identity and old-writer state intact while the storage operator follows the driver’s supported recovery. Do not delete a PVC/PV, remove finalizers, clear claimRef, force detach, format a device, or edit a node mount as a generic fix. Delete reclaim policy can remove the underlying storage after object deletion; Retain also is not a backup.
Changing a workload/class or replacing a volume is a separately reviewed mutation with data backup, writer/quiescence and restore/cutover constraints. Roll back a failed approved configuration change through its owning GitOps/deployment workflow only if the original placement/data relationship remains valid. A recovered mount does not reverse a data migration or schema change.
When the observed state is incomplete
Use the diagnostic for the specific uncertainty. No branch changes resources or grants permissions. Preserve the initial object UID and failure window so a newly created replacement is not presented as recovery of the original volume.
Forbidden or missing storage APIs
Check the selected identity’s permission and API discovery. A forbidden result is not an absent driver. Request only the needed observation from the storage owner; do not broaden the role to make a diagnostic pass.
k auth can-i get persistentvolumes
k auth can-i get storageclasses.storage.k8s.io
k api-resources --api-group=storage.k8s.ioClaim remains Pending with no selected node
Recheck the real consumer and scheduling conditions. WaitForFirstConsumer may be working as designed when no consumer or eligible placement exists; it is not evidence for deleting the claim.
k get pod 'example-app' -n 'apps' -o jsonpath='{.spec.nodeName}{"\n"}{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}{"\n"}{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'
k events -n 'apps' --for 'pod/example-app'Bound but Pod still cannot start
Reconfirm the claim/PV identity and Pod events. Bound is earlier than attachment and mounting. Take the exact stage and node to the driver owner rather than attaching another writer.
k get pvc 'data-example' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.status.phase}{"\n"}{.spec}{"\n"}{.metadata.annotations.volume\.kubernetes\.io/selected-node}{"\n"}'
k events -n 'apps' --for 'persistentvolumeclaim/data-example'
k events -n 'apps' --for 'pod/example-app'Events have already expired
Read the current object UID/conditions and retain the evidence gap. Use existing central events/controller logs with the owner. Do not delete/recreate objects just to generate a fresh error for this article.
k get pod 'example-app' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.creationTimestamp}{"\n"}{.status.conditions}{"\n"}'
k get pvc 'data-example' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.creationTimestamp}{"\n"}{.status.phase}{"\n"}'LET’S BUILD
Verify one consistent storage relationship and the application
After a separately approved fix, repeat the Pod/claim/PV relationship and event observations. If a controller replaced the Pod, record its new UID and actual node and update the public examples deliberately. Do not reuse the old node assumption for the new instance.
Accept recovery only when expected binding, any required attachment and node mounting complete, the replacement/retained workload reaches its expected health, and its owner confirms the existing data and application consistency checks. A bound claim or a successful empty test file is not a recovery test for real data. These read-only panels require no rollback or new cluster cleanup.
Same authorized Bash shell on the administration host
k get pod 'example-app' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.metadata.ownerReferences}{"\n"}{.status.phase}{"\t"}{.spec.nodeName}{"\n"}{range .spec.volumes[*]}{.name}{"\t"}{.persistentVolumeClaim.claimName}{"\n"}{end}{range .status.conditions[*]}{.type}{"="}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
k events -n 'apps' --for 'pod/example-app'
k get pvc 'data-example' -n 'apps' -o jsonpath='{.metadata.uid}{"\n"}{.status.phase}{"\n"}{.spec}{"\n"}{.metadata.annotations.volume\.kubernetes\.io/selected-node}{"\n"}'
k events -n 'apps' --for 'persistentvolumeclaim/data-example'The expected workload can use the correct existing data relationship, with actual identity changes and remaining visibility limits recorded.
GO TO THE SOURCE
Official references
Each environment states how it was reviewed. A documentation review does not claim a hands-on test of every version.
- Kubernetes maintained releases
- kubectl/API-server version skew
- Official Linux kubectl setup and verified binary workflow
- Kubeconfig context and trusted access
- Kubernetes authentication plugins
- Kubernetes authorization checks
- Ubuntu lifecycle
- Debian release lifecycle
- RHEL supported lifecycle
- Rocky Linux release branches
- AlmaLinux releases
- Manjaro complete upgrades
- PersistentVolume lifecycle, binding, access modes and reclaim policy
- StorageClass binding mode and topology
- CSI architecture, sidecars and driver-specific compatibility
- CSIDriver attachRequired and mount policy
- CSINode registration and topology keys
- VolumeAttachment API status
- Dynamic provisioning
- Debugging Pods
- Kubernetes events
KEEP LEARNING