Incident March 30th, 2026 – Accidental CDN Caching

stingraycharles

This write up doesn’t make sense. Authenticated users are the ones without a Set-Cookie? Surely the ones with the cookie set are the authenticated ones? There are dozens of contradictions, like first they say: “this may have resulted in potentially authenticated data being served to unauthenticated users” and then just a few sentences later say “potentially unauthenticated data is served to authenticated users” which is the opposite. Which one is it? Am I missing something, or is this article poorly reviewed?

sublinear

I'm curious if having unique URLs per user session would mitigate this. I think that's already best practice in most API designs anyway?

sebmellen

Almost three years ago now, Railway poached one of our smartest engineers. They were smart to do so. I have a lot of respect for the Railway team and I’m impressed with their execution. I think this is their first major security incident. Good that they are transparent about it. If possible (@justjake) it would be helpful to understand if there was a QA/test process before the release was pushed. I presume there was, so the question is why this was not caught. Was this just an untested part of the codebase?

varun_chopra

The status page [1] has the actual root cause (enabling "Surrogate Keys" silently bypassed their CDN-off logic). The blog post doesn't. That's backwards. "0.05% of domains" is a vanity metric -- what matters is how many requests were mis-served cross-user. "Cache-Control was respected where provided" is technically true but misleading when most apps don't set it because CDN was off. The status page is more honest here too: they confirmed content without cache-control was cached. They call it a "trust boundary violation" in the last line but the rest of the post reads like a press release. No accounting of what data was actually exposed. [1] https://status.railway.com/incident/X0Q39H56

muragekibicho

Does Stripe use Railway? The dashboard was down today and this is the only incident report I've encountered and the timeline matches Stripe's downtime.

rileymichael

pretty hard to find this on their blog, looks like incidents are tucked away at the bottom. an issue of this size deserve a higher spot. (also looks like two versions of the 'postmortem' are published at https://blog.railway.com/engineering )

heyethan

Caching is one of those systems that works perfectly — until it amplifies the wrong thing. Feels like you have to define what’s safe to cache before optimizing for speed.