WIUNIX ACADEMY
TCP stalls on Linux: diagnose PMTU black holes and retransmissions
Page 2 of 3. Review the earlier steps and prerequisites before continuing.
YOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment6 editable values
Separate PMTU, MSS and retransmission evidence
PMTU is the IP-packet size the sender currently believes the path can carry. MSS concerns TCP data, so IP/TCP headers and options matter; a peer’s advertised MSS is not a path measurement. ss reflects the local endpoint’s state, which can be cached or later adjusted. It is not a measured minimum MTU of every router.
In the current iproute2 implementation, retrans is printed as outstanding/total retransmissions when available. The lifetime total is useful only for the same socket, and bytes_retrans may be exposed separately. Compare counter growth with forward progress and timeouts rather than reading a historical nonzero total as an ongoing fault.
A PMTUD black-hole hypothesis becomes stronger when small exchanges work, a larger transfer repeatedly stalls, retransmissions grow without useful progress and size/ICMP evidence agrees. A single lost ping or one retransmission is insufficient. Even a TLS handshake can stall on a large certificate flight, so the visible stage alone does not isolate TLS configuration from the network.
LET’S BUILD
Optional: compare small and larger DF probes
Only probe the agreed endpoint in the permitted window. These are at most six synthetic ICMP requests, not customer data or a load test. ping -s is ICMP payload bytes; for ordinary IPv4 without IP options add 28 bytes for IP and ICMP headers. The selected payload 1372 therefore is not itself a TCP MSS or a proven path MTU.
Small replies with consistent larger failures suggest a size-dependent path or ICMP policy difference, not proof that TCP uses the same route. A local message-too-long/mtu error may come from the interface or cached PMTU before transmission. No replies at either size may simply mean ICMP is blocked; do not lower MTU from that result alone.
Authorized client-host terminal in the affected network namespace
date -u +%FT%TZ
ping -4 -n -I '192.0.2.10' -M do -s 56 -c 3 -W 1 -w 5 '198.51.100.20'
ping -4 -n -I '192.0.2.10' -M do -s '1372' -c 3 -W 1 -w 5 '198.51.100.20'Record replies, local errors and timeouts separately for both sizes. The result supports or weakens a hypothesis without changing any MTU.
LET’S BUILD
Optional: collect a separate UDP path clue
tracepath uses UDP and may vary destination ports. Run it only if that probe traffic is permitted to the same endpoint. The initial packet length here is 1500 bytes and the hop limit is 12; timeout bounds the observer’s duration. This is a separate probe family, not a TCP-port test.
A pmtu report is a clue from this UDP/error-queue exchange. This tracepath panel cannot bind the source address or interface chosen in this form; the kernel may select a different source or route. Different ECMP routing, ICMP filtering and the return path can make it disagree with TCP or ping. An incomplete trace or a timed-out hop is not automatically packet loss affecting the application; routers may rate-limit replies while forwarding traffic normally.
Authorized client-host terminal in the affected network namespace
timeout --signal=INT 15s tracepath -4 -n -l 1500 -m 12 '198.51.100.20'Retain a bounded UDP path/PMTU clue or an explicit incomplete result; do not label it the measured TCP path.
LET’S BUILD
Optional: observe only the fragmentation-needed header
Use this narrow observer only with approved capture access, on the confirmed client interface, while an already authorized TCP attempt runs in another terminal. It accepts IPv4 ICMP type 3/code 4 messages addressed to 192.0.2.10 whose quoted IP protocol is TCP. It does not capture ordinary TCP data packets.
The fixed 68-byte snapshot is deliberately too short to include application data from a valid quoted ordinary IPv4 TCP packet, even before accounting for link headers. It can truncate quoted transport details; that is expected. No packet file, ASCII/hex payload dump or promiscuous mode is used. Keep even endpoint/port metadata private and do not increase the snapshot to investigate private traffic.
A matching need-to-frag/MTU message shows that such feedback reached this observation point. It does not prove the kernel accepted it or that every direction is healthy. Silence can mean no oversized packet, a different interface/namespace, capture loss, filtering or a black hole; it cannot establish which one. The timeout normally stops a quiet observer with a timeout exit status.
Authorized client-host terminal in the affected network namespace
sudo timeout --signal=INT 15s tcpdump -i 'eth0' -p -nn -v -s 68 -c 5 'icmp and dst host 192.0.2.10 and icmp[0] = 3 and icmp[1] = 4 and icmp[17] = 6'At most five narrowly filtered ICMP error headers, or an explicit quiet/truncated/permission result, are observed without a packet file or application payload.
LET’S BUILD
Optional: identify a small HTTPS request’s failing stage
Use this only for an existing public, harmless endpoint that permits HEAD and direct access. It makes one request to probe.example.net on 443, pinned to 198.51.100.20 and bound to local address 192.0.2.10, while retaining the hostname for TLS verification and SNI. It sends no credentials or cookies and prints no response headers/body. The ephemeral source port, ECMP choice and packet marks still need not match the real application flow. If organizational policy requires a proxy, skip this direct probe rather than bypass that policy.
The first -q disables a user curl configuration; the empty proxy option makes the intended direct path explicit. Certificate verification stays enabled. The reported times are cumulative milestones, not independent phase durations. A HEAD response is a small exchange and does not prove bulk transfer health. A certificate validation error needs TLS diagnosis; a handshake timeout can still have a network cause. Keep the curl exit error beside the timings, since http=000 alone is inconclusive.
Authorized client-host terminal in the affected network namespace
curl -q --proxy '' --interface '192.0.2.10' --http1.1 --head --silent --show-error \
--resolve 'probe.example.net:443:198.51.100.20' \
--connect-timeout 5 --max-time 15 --output /dev/null \
--write-out 'local=%{local_ip} local_port=%{local_port} remote=%{remote_ip} http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}\n' \
'https://probe.example.net:443/'The direct address, curl error and cumulative connection/TLS/first-response milestones are known for one authorized HEAD request; bulk TCP remains a separate question.
Choose the next owner and the smallest justified change
Size-dependent stalls plus retransmission/no-progress and corroborating MTU feedback support investigating a tunnel/egress MTU mismatch or blocked valid ICMP feedback. Take the real flow tuple, namespace, direction and timestamps to the network owner. Broad loss across sizes with interface/backend congestion evidence needs a loss/capacity investigation instead; retransmission alone does not distinguish them.
If TCP acknowledges application data normally while the application waits, investigate its server-side work and dependencies. If certificate/hostname verification fails, use the separate TLS guide without disabling verification. Do not change Nginx buffering, QUIC settings or TLS ciphers to repair an unproven IP-path problem.
Any actual fix is a separate scoped change on the owning layer: for example correcting an identified tunnel overhead calculation or permitting the required, validated ICMP feedback through the existing firewall workflow. Record the previous configuration, console recovery path and affected traffic first. Do not globally disable PMTUD, enable probing, clamp MSS to a copied number, flush route metrics or lower every interface MTU as a generic remedy.