Security Disclosure

Keloran

Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not

MattIPv4

Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.

MattIPv4

Related: https://news.ycombinator.com/item?id=50177413

theturtletalks

Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up: https://x.com/wickedguro/status/2113015432974405658

DrawTR

Jeez, what a rough past 24 hours for all security folk. Hang in there

paxy

It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.

wateralien

Is this our new AI daily reality?

esseph

Fuck, I thought this was about trains

minraws

how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture. Like did someone vibe code too hard, how do env secrets leak for everyone...

gbnwl

The past two days have been awful as someone who uses Railway.

martinald

I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches. Fwiw it sounds similar to the hack over the weekend that opencode suffered.

lijok

Damn I guess all those l33t hiring practices were for nothing

alpineman

Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.

AJRF

Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?

eecks

Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36

sergiotapia

If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.

fbrncci

Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…

fyzix

How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...

shinycode

I’m lucky I moved away from it to host my own db with coolify

vldszn

Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway. And it’s all just getting started imo. https://blog.cloudflare.com/emudhra-certificate-compromise https://www.wiz.io/blog/php2shell-php-rce