Degraded availability for GitHub Actions, GitHub Pages and Webhooks

defenestration

GitHub actions staying 'Queued', started 16 minutes ago for me

EduardoBautista

I am going to try out Cursor Origin, GitHub is just too much of a bottleneck these days. The only thing that is not readily available is integration with claude code, but with the new claude code mods feature I think it's something I could vibe code quickly.

outworlder

At my company, we migrated basically all CI/CD to Github Actions. Also our Gitops processes. I have warned that putting all the eggs in one basket and they key to the fridge in the same basket was a bad idea. People didn't care. So I don't care anymore. Github seems to have degraded measurably after they got acquired. As is tradition.

serial_dev

I find it baffling that I knew about the incident 20 minutes before the official status update…

esafak

To all those in the middle of conjuring a Github-killer into existence, more power to you, and please ship it already; I can't take this any longer.

willio58

I wonder if this will ever end (meaning they've solved scaling for AI), and if not, is that due to the problem being unsolvable at this scale or bad decisions on GitHub's part? For example, they always bring up AI as a cause, but it's also apparent the move to Azure didn't help.

icameron

I’m going to hope they can figure things out and improve their uptime. It’s so engrained in my teams workflow at this point, tearing it out and switching to anything else would be extremely disruptive. I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are

somehowadev

It's almost a running joke at my company that around 15:00 UTC we're going to get the weekly Github actions outage. I'm assuming it's something to do with the USA users getting their first work in?

FLeXMurphy

It would not surprise me if they are going to shift to Plan B sometime soon, because this isn't working. They can't keep provisioning hardware resources in perpetuity to keep up with volume that the AI is pummeling them at. I think most other repo providers have either experienced downtime and/or have locked themselves up to prevent the onslaught. It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.

rvz

Last time this happened was a week ago. [0] There’s no CEO of GitHub to contact and Microsoft does not care as Copilot, and Tay continue to wreck the platform as the incidents continue. No better time to self-host, since as this point your self hosted setup would be far most stable than GitHub as predicted 6 years ago. [1] [0] https://news.ycombinator.com/item?id=50057012 [1] https://news.ycombinator.com/item?id=22868406

kingcauchy

I know there's a lot of GitHub annoyed people out there but after investigating a lot of the instances of outages it does seems like A) They're abusing ElasticSearch in a way is was never meant to be used B) They've seen an unprecedented increase in scale and now have to provide agent slop a lot of free action minutes until they figure out how to separate commercial support from free support C) Who knows how many other systems were never meant to see this scale there Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.

haute_cuisine

Their infra is under constant DDoS + they provide a lot of free stuff for open source projects. There's also a constant churn of dependencies, patches, abandoned libraries, etc. We should give them some appreciation, dealing with this stuff is hard.

JohnMakin

They should just call it Monday now

cwyers

There is a screamingly obvious answer here and I don't understand why they aren't taking it -- raise prices, make the free tier less generous. Either slow down the growth or up the revenue pipe so you can throw more compute at it.

renehsz

https://GiveUpGithub.org - why and how to ditch GitHub.

efficax

github has basically become a joke at this point. everyone engineering org that i know of is at least asking themselves if there's a migration path off there if they're not in the process of building one. Every slack has a custom emoji of github on fire. Github is so joever

minraws

Can we split Github, paid/pro users, my orgs are payings north of 25k per year to Github, can we get better service please, even AWS assigns a proper support performance once you hit few tens of thousands in spend. This is just insane, that all of github starts failing at once, I would like to have better experience for when I am working compared to when I want to spam commits with ai to slop something.

figassis

I'm wondering if they can just build an entirely new platform, onboard new users to it, and then slowly migrate old ones. Even if they don't migrate, the load would stabilize. What are the cons here?

Rantenki

I am surprised that this is still newsworthy. It seems like _something_ is down at Github nearly every day, and their status history backs up that assessment: https://www.githubstatus.com/history If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.