Three Days in August: What a DDoS Attack Exposed in Our Network

nine_ch

We're a Swiss managed hosting provider and run our own network. In August one of our customers was hit by a UDP amplification attack that we estimate peaked at 500 to 600 Gbit/s across all our links combined, several times what our uplinks can carry. Because we provide that customer's internet connection, the impact hit our network directly, and roughly three hours later the attack was broadened to our own services as well. The write-up is mostly about what we got wrong: our automated detection only covered our own prefixes, not customer prefixes we announce on their behalf. Blackholing at the IXs we connect to only took effect via route servers, not on direct peerings. And we had no way to withhold a prefix from one specific upstream, so we had to build that while under full load. On top of that, our own website runs on the same shared platform as customer apps, so when it was targeted, unrelated applications were affected too. What eventually ended it was moving exposed applications behind a CDN with DDoS protection. The full timeline is in the PDF postmortem linked from the post. Happy to answer questions.

majke

8. 80.239.216.210. 0.0% 78 123.2 64.6 42.9 141.8 30.6 9. vl202.zur-itx1-dist-1.cdn77.com. 0.0% 78 60.1 58.8 43.7 136.5 24.2 10. 89-187-165-194.bunnyinfra.net. 0.0% 78 45.9 63.5 41.8 131.9 31.0 so cdn77.com and bunny.net

jareklupinski

> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion. "AI said it's all good. There are no attackers within our walls."

tshanmu

"We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them." claudism?

someonebaggy

Did AI write this?

BLKNSLVR

Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type? In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack? If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic? Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years? I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?

cube00

> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion. Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra. > Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained. I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?