Wiunix
Talk to an Engineer
← Start of tutorial

WIUNIX ACADEMY

Linux says disk full: distinguish blocks, inodes and deleted open files

Page 2 of 2. Review the earlier steps and prerequisites before continuing.

YOUR SYSTEM. YOUR STEPS.

Where are you working?

Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.

Ubuntu 24.04 LTSReviewed against official documentation ·

USE YOUR OWN VALUES

Configure this tutorial

Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.

Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.

Set values for my environment4 editable values
05

LET’S BUILD

Find unlinked files that still have an owner

List open files with link count below one. Match the DEVICE and NAME to the affected filesystem, then inspect the PID and descriptor. A deleted marker is not permission to truncate: it could be a database segment, a memory-mapped object or an application’s recoverable state. Lsof can return nonzero when nothing matches; keep diagnostics and distinguish that from lack of permissions.

bash

Authorized Linux terminal

sudo lsof -nP +L1
How do I know it worked?

For each significant retained file, record PID, FD, device, inode and allocated-size evidence privately; do not sum repeated descriptors as distinct files.

06

LET’S BUILD

Map a measured PID to the owning service

Replace 1234 with the PID observed in the previous step immediately before use. PIDs can be reused, so reconfirm the command and cgroup; inspect only the needed identity fields rather than dumping environments that may contain secrets. Do not kill a process just because it holds a large file.

Use the observed numeric descriptor 5, without lsof’s mode letter. SIZE/OFF is logical size or offset, not allocated bytes. Stat allocation is blocks multiplied by block_bytes; deduplicate by device and inode. Shared extents, snapshots and filesystem-native accounting can make this differ from uniquely reclaimable physical capacity. Reconfirm PID/FD immediately; a disappearing descriptor means repeat ownership discovery, not guess a new number.

bash

Authorized Linux terminal

ps -p '1234' -o pid=,ppid=,user=,comm=,lstart=
sudo cat '/proc/1234/cgroup'
sudo lsof -nP -a -p '1234' +L1
sudo stat -L -c 'type=%F device=%d inode=%i size=%s blocks=%b block_bytes=%B links=%h' '/proc/1234/fd/5'
How do I know it worked?

The same PID/FD still identifies the target inode, with logical size and allocation reported separately.

Reclaim through the owner, not through a file descriptor

For a confirmed disposable log, ask the owning daemon to reopen it using its documented mechanism after rotation; for Nginx that is USR1/reopen, not an arbitrary full restart. For other state, schedule the application’s graceful stop/restart with recovery checks. Do not truncate a proc descriptor, remove database files, clear every cache or change reserved blocks as a generic fix. Those shortcuts can destroy evidence or corrupt a running application.

07

LET’S BUILD

Measure the journal before choosing a retention change

If the journal is a measured contributor, inspect its own disk accounting. Any vacuum operation removes archived history and must follow the agreed retention and incident-evidence requirements. Do not invent an arbitrary size cap in the middle of an incident. Application log files and journal accounting are separate contributors.

bash

Authorized Linux terminal

sudo journalctl --disk-usage
How do I know it worked?

Journal usage is measured; no history is deleted by this diagnostic.

08

LET’S BUILD

Prove both capacity and application recovery

After the owner-approved native reclamation or expansion, repeat the same filesystem measurements and the deleted-open check. A reduced du total without increased df availability may mean the process still holds the inode. Free space alone does not prove that failed writes or application transactions have recovered; run the owning application’s bounded health/write-read check.

bash

Authorized Linux terminal

df -hT -- '/var'
df -i -- '/var'
sudo lsof -nP +L1
How do I know it worked?

Available capacity/inodes improve as expected and the owning application’s verified operation recovers; unmatched files remain explicitly unresolved.

Prevent the same failure mode

Monitor both byte and inode headroom, growth rate and backup/restore requirements. Verify log-rotation reopen hooks and application-owned retention. On snapshotting or thin-provisioned storage, alert on the native allocation pool too. The investigation is documented, not a hands-on filesystem repair claim; all cleanup decisions remain tied to a concrete owner and recoverability.

GO TO THE SOURCE

Official references

Each environment states how it was reviewed. A documentation review does not claim a hands-on test of every version.

Explore all tutorials ↗