GitHub having issues [resolved]

joecool1029

codeberg might be a little slower on git cli, but at least it's not becoming a weekly 'URL returned error: 500' situation...

joshrw

Happening very often lately

duckkg5

I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability

cpfohl

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green. But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

akoumjian

Is this related to Cloudflare? I'm getting cf-mitigated: challenge on openai API requests. https://www.cloudflarestatus.com/ https://status.openai.com/

garciasn

How reliable is githubstatus.com? I know that status pages are generally not updated until Leadership and/or PR has a chance to approve the changes; is that the case here? Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g. Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available. But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))

khaledh

GitHub has been shit lately. What the fuck is going on?

overshard

I've taken to hosting everything critical like this myself on a single system with Docker Compose with regular off premises backups and a restore process that I know works because I test it every 6 months. I can swap from local hosting to a VPS in 30 mins if I need to. It seems like the majority of large services like GitHub have had increasingly annoying downtime while I try to get work done. If you know what you're doing it's a false premise that you'll just have more issues with self hosting. If you don't know what you are doing it's becoming an increasingly good time to learn. I've had 4 years of continuous uptime on my services at this point. I still push to third parties like GitHub as yet another backup and see the occasional 500 and my workflow keeps chugging along. I've gotten old and grumpy and rather just do it myself.

cyberax

You know that it's bad when the status page doesn't have the availability stats anymore.

pothamk

What’s interesting about outages like this is how many things depend on GitHub now beyond just git hosting. CI pipelines, package registries, release automation, deployment triggers, webhooks — a lot of infrastructure quietly assumes GitHub is always available. When GitHub degrades, the blast radius is surprisingly large because it breaks entire build and release chains, not just repo browsing.

littlestymaar

In many companies I worked for, there were a bunch of infrastructure astronauts who made everything very complicated in the name of zero downtime and sold them to management as “downtime would kill pur credibility and our businesses ”, and then you have billion dollar companies everyone relies on (GitHub, Cloudflare) who have repeated downtime yet it doesn't seem to affect their business in any way.

banga

Only on days with a "y"...

m_w_

Seems like the xkcd [1] for internet infrastructure that was posted earlier [2] should have github somewhere on it, even if just for how often it breaks. Maybe it falls under "whatever microsoft is doing" [1]: https://www.reddit.com/r/ProgrammerHumor/comments/1p204nx/ac... [2]: https://news.ycombinator.com/item?id=47230704

zthrowaway

Microslop ruins everything it touches.

fsflover

Dupe: https://news.ycombinator.com/item?id=47237018

yoyohello13

How many 9s is GitHub at now? 2?

granzymes

I have a bug bash in an hour and fixes that need to go in beforehand. So of course GitHub is down.

shykes

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages. I'm biased because we (dagger.io) provide tooling that makes this break-glass setup easier, by decoupling the CI logic from CI infrastructure. But it doesn't matter what tools you use: just make sure you can run a bootstrap CI pipeline from your local machine. You'll thank me later.

Imustaskforhelp

Are we serious?

terminalbraid

I would prefer we have posts when github is not having issues to cut down on noise.