The iPad was on Tailscale: a WebRTC debugging story

syllogistic

Author here. This started as a blank page on one device and ended two weeks later at the intersection of two bugs: webrtc-rs hardcodes INITIAL_MTU=1228 [never updated, no path probing, retransmits at the same size forever], and Tailscale's packet filter classifies any IPv6 packet with a Fragment header as unknown protocol, so the default deny fires. On every platform, counted under reason="acl". Neither is unreasonable alone. Together: silent wedge, every health check green, because everything that tests the path is small and only the payload fragments. Two-command repro on any tailnet: ping -s 100 works, ping -s 1400 over the Tailscale IPv6 address is 100% loss. Full WebRTC repro and captures: https://github.com/phact/mtu-webrtc-bug . We've reported upstream to both projects https://github.com/tailscale/tailscale/issues/20083 and https://github.com/webrtc-rs/webrtc/issues/806 . Happy to answer questions. Especially interested if anyone knows the history behind the IPv6 fragment decision in Tailscale's filter.

inigyou

I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets. It is impossible to parse the UDP or TCP port number out of a fragment. This is surely the reason the ACL module entirely rejects them. TCP will adjust it's segment size based on PMTUD so as to not require fragmentation. This is why it hasn't been noticed so far. But fragmented UDP packets are a corner case of normal behavior and it boggles the mind that someone could just decide to completely drop them. UDP fragment filtering could be implemented by a global fragments on/off setting (works for "allow everything" = fragments on, cautious = fragments off) or by blocking the first fragment which includes the port number (and blocking it if the port number is split across fragments which I think is technically allowed but completely abnormal).

katericksonnow

MTU black holes are the worst because every health check is small enough to survive.

hylaride

I'm having flashbacks to 1990s-era PPPoE, where the slightly smaller MTU had issues with some server OS's that had TCP/IP stacks that didn't support or ignored MTUs smaller than 1500 bytes and bulk data transfers would get messed up. I don't remember which ones, but it was some commercial UNIX.

Sean-Der

Amazing debugging, I loved reading that. HN doesn't get enough good posts like this anymore :) If https://github.com/pion/sctp/issues/12 had happened (not just in Pion but across all implementations) this could have been fixed years ago. The hardcoding we all settle for is tragic.

Veserv

Ah yes, the horrible anti-feature of IP fragmentation strikes again. Pair it with the anti-solution of dropping large packets instead of truncating them and we get our perfect storm of bad design that is MTU incompatibility and modern MTU discovery.

cyberax

Another fun happy iOS story: we were launching our app a year ago, with a self-imposed deadline. As usual, tons of bugs were being fixed in the last moment. And then our authentication stopped working on simulated iOS devices (while still working on the real devices!). After hours of frantic debugging and staring at Wireshark dumps, I found the issue: HTTP3 and QUIC. Apparently, the simulated stack was not tracking the MTU correctly and was trying to send 1506-byte UDP packets. The "fix" was to add deny rules for UDP ports 80/443 to our firewall.

OptionOfT

One day I hope to work on problems like this. Fantastic article.

talkingtab

Hunting good bugs is something every good software developer should experience. A good interview question is "tell me your favorite bug". Bugs are about reasoning, not intelligence. And I will take someone who can tell me what is wrong over what is correct any day. It requires a focus on getting things actually work. I have two favorite bug stoies. The first is from a printout from the run of an IBM 360 assembly language program when I was just learning. Someone asked em why their program failed to run. I glanced quickly at the front page of the printout and it said "Too Long". So I told the person that was the problem. Something was too long. He looked at me very strangely, so I looked back at the page a little more closely, only to notice "Too Long" was in the name field of the person running the program. He was Vietnamese and his name was Too Long - literally. There is a powerful lesson (at least one) there. The other happened when I was implementing some AppleTalk protocols - NBP to be exact. (Don't ask). I would capture the working packets then compare all the checksums, headers, constants, length fields in the packet my code generated and fix any problems. I was stuck on one failure. I just could not see any difference as I went through byte by byte, time after time. It was late and time to go home so I decided to print off each packet on paper and compare them later - certain I was missing something. The problem was instantly obvious. One printout took a page, the two pages. I had been appending junk data in the packet. Sigh

torlok

Reading titles like these makes me feel like I'm having a stroke.

syllogistic

PR for the webrtc-rs half is up! https://github.com/webrtc-rs/webrtc/pull/807

bastard_op

Developers should spend more time being network engineers before writing network code. I saw this title and my first thought was, "What, mtu?" If you've ever had to support vpn's in an enterprise securing businesses with ipsec or sslvpn with tunnel overhead, you've run into mtu issues. Some apps/protocols or firewalls misbehave, devs/engineers didn't read the memo from 20 years ago in rfc form how ipv6 mtu's work (and missed v4 to boot, lucking out with 20 more years of someone else fixing it). Not Tailscale or Cisco or in between are immune to mtu issues in vpn or networking.