Skip to content

Double VPN & Vanishing TCP: Fixing MTU and MSS Blackholes

Your tunnel is up. DNS resolves, ping works, small requests fly through — yet the moment a browser opens an HTTPS site, it stalls after the first couple of kilobytes and times out. Meanwhile, UDP traffic flows without a hitch.

Whenever two tunnels are stacked in series (for instance, OpenVPN/Pritunl on the outer leg feeding a second L3 relay tunnel downstream), 90% of the time the culprit is TCP MSS exceeding Path MTU.

Why It Happens

During the three-way TCP handshake (SYN / SYN-ACK), each endpoint announces its Maximum Segment Size (MSS) derived from its own outbound interface MTU:

MSS = MTU − 40  (20 bytes IPv4 + 20 bytes TCP)

With a single well-configured tunnel, the VPN interface lowers its MTU to account for encapsulation overhead, MSS shrinks accordingly, and packets fit cleanly.

With two nested tunnels, the inner tunnel or server behind it often advertises MSS = 1380 (for MTU = 1420) without knowing anything about the outer tunnel’s overhead (or an underlay cloud NIC running at MTU = 1450). Adding TCP timestamps (+12 bytes) and IPv4 headers (+20 bytes) produces 1412–1432-byte packets that exceed the outer tunnel’s true payload ceiling.

If ICMP “Fragmentation Needed” (Type 3, Code 4) messages make it all the way back to the sender, Path MTU Discovery lowers segment size automatically. Across double NAT and nested tunnels, however, ICMP errors are frequently dropped or stripped. Small packets (SYN, ACK, DNS) still pass, while the very first full-sized segment — a post-quantum TLS 1.3 ClientHello, a server certificate chain, or an HTTP body — vanishes silently into a PMTU blackhole.

Confirming in 30 Seconds

Check interface MTUs on the routing gateway (ip link show), then run unfragmented pings (DF bit set via -M do) from the client:

# Drops if Path MTU is smaller than 1428 bytes:
ping -M do -s 1400 <target-ip>

# Succeeds if ceiling lies between 1328 and 1428 bytes:
ping -M do -s 1300 <target-ip>

Inspect negotiated MSS on live sockets on the server:

ss -tipn | grep mss

The Fix: Clamp TCP MSS in the Handshake

Rewrite the MSS option inside SYN / SYN-ACK packets traversing the forwarding gateway using iptables TCPMSS:

iptables -t mangle -I FORWARD 1 \
  -i <inner-tunnel-iface> \
  -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1280

At MSS = 1280, the resulting IP packet (1280 + 32 TCP/timestamps + 20 IP = 1332 bytes) clears virtually any nested OpenVPN, WireGuard, or cloud VXLAN underlay with comfortable headroom, while adding negligible header overhead (~7% vs. 1380).

Remove cleanly on tunnel teardown:

iptables -t mangle -D FORWARD \
  -i <inner-tunnel-iface> \
  -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1280

(Note: when interface capture capture_iifs is used in ololo-relay, TCPMSS 1280 clamping is applied automatically on transit flows.)