Software is breaking down at a systemic level

zero_shift 356 points 223 comments November 16, 2026
www.breck-mckye.com · View on Hacker News

Discussion Highlights (20 comments)

tcoff91

There were package managers long before npm. Maven, rubygems, etc...

dividedcomet

This feels a little hyperbolic, the grounding was a result of a cable swap and the ssh cert incident they cite was so human caused it’s not even funny. I think things are a little more shaky, but not massively so. I had the town I was living in during college lose its internet and phone lines due to a cable being cut on a mountain pass and it was fine. If your business is sensitive to an email outage, prioritize high reliability over integration advantages that outlook offers. The php2shell vuln is bad, but we also had log4j years ago.

LPisGood

Software has always been broken at a systemic level. It’s a complicated continuously changing web of abstractions and abstraction breaking hacks attempting to survive prolonged contact with reality.

herval

This reads like nostalgia way more than reality

jmclnx

Well there was this saying "Do One Thing and Do It Well", seems once again we are seeing the affects of ignoring that.

anonreplier

There are some decent points, but this is collating a lot of separate incidents with wildly different causes that just happened to occur in a close timeframe.

drooby

Most folks here didn't even know, but a Bitrise security outage hobbled our entire CI this weekend. Friday to Sunday night. The security issue motivated all Bitrise customers rotating their secrets. this was a big deal for mobile deployments but didn't get much attention on HN..

rurban

He obviously is preparing another doomsday conference talk. Nothing interesting to see here. On the other hand software is improving dramatically at a systemic level, even without safe system languages arriving on the horizon.

AndrewKemendo

Software was mostly always broken but it was hidden and invisible, now it’s visible because the cost of making it visible is so low This is classic surveillance/detection bias: https://healthjournalism.org/glossary-terms/surveillance-bia... It’s basically free to black/grey/whitehat penetration test anything and everything without having to even know what ghidra or metasploit so everything gets penetrated

jstimpfle

My impression is it's always been a mixed bag. The level of bugginess has been relatively constant. In the meantime, scope and capabilities have gone way up. I hate Teams and Outlook and the virus scanner on my company's laptop though. Those are keeping me from doing my work. Windows + anti-virus makes file access and process spawning 10x to 100x slower compared to a Linux machine.

adamddev1

People keep believing that the big LLM players with somehow disprove Rice's Theorem. "Somehow we'll be able to get AI to check if a program does X or Y." "We'll be able to have the machines verify things automatically." But this is is of course as hopeless as solving the Halting Problem. Theoretically, we know this is impossible. Empirically, we can observe the breaking effects of this broken thinking. Code quality goes down, and apps go from being 99% to 95% reliable. But that's not a big deal, they say, that's good enough... But when the quality of all the building blocks further down get degraded in the same way, you can't keep track of which parts are broken, and which parts are dependable, and the whole thing comes crumbling down. People keep saying, "I can build an app with AI, therefore we can build everything with AI." This article talks about some potential anecdotes that could help to show that doesn't work.

tracerbulletx

When I first started my career we used to ship like once a month and that was considered good, everything was extremely convoluted and worse than it is today, whole segments of the industry didn't even use Version Control, half the government websites you used stored passwords in the clear and emailed them to you for resets, e-commerce was a cgi script that sent an email for fulfilment and it took weeks to get your shipment, sites went down at night and weekends for maintenance regularly, servers fell over with traffic that would run on a raspberry pi now, but for some reason everyone keeps saying how bad things are now. They're delusional or don't actually remember the past.

SoftTalker

> You can find programmers who know how to manage the complexity, but they want to be paid lots of money. This wasn't always the case, by the way. It started with the "dot-com" mania and VC investment in the late 1990s. Before that, a programming job paid decently well but it was on the level of other cost-center professional jobs such as accounting. A comfortable living, but nothing extravagant.

spiralpolitik

There have been parts of the software industry that have been mission critical from day one and have built processes, languages, and tools to make sure things are 100% reliable. Building software that way is slower and more expensive and in the past has been an overkill for most systems. The problem software engineers are facing is that now that software is ubiquitous, almost everything needs to be 100% reliable (or close to it). This is coming into conflict with the move fast and break stuff everything needs to be done this quarter at minimal cost mentality. Hence why we are at the breaking point.

travisgriggs

> About ten years ago, two small revolutions happened in our industry ... And then, we had the Open Source boom. This seems a bit myopic. Or naive. Open Source "happened" a lot longer than 10 years ago. Microsoft didn't shit itself about Linux just 10 bare years ago. And it's not like prior to 10 years ago, the narrow focus of npm world wasn't cobbling web sites together with bits and pieces garnered from wherever. That said... > But AI is both a cause and a symptom This I agree with 100%. The rise of AI and necessity of AI to continue to make progress is a massive indictment against an industry that prioritized more features and uptime over solid first principles for years, taking out promisory note after note in complexity, in hopes that the stuff papered over would quietly just fade away.

TacticalCoder

It's because amateurism has been hailed as the second coming of Christ. Leftpad, mentioned in TFA, is a case in point: not because of leftpad itself but because of all those defending leftpadism. "People can't write a proper leftpad" . Yeah, well... If a dev cannot write a proper leftpad, then it should never have been a dev in the first place . Someone who cannot write leftpad is not qualified to use leftpad as a dependency (and, no, leftpad is neither the Linux kernel nor a cryptographic libray: it's fucking leftpad, which is the entire point). We've had "devs" whose only qualification was writing glorified "punch the monkey" ads in JavaScript: the JavaScript surveillance/ad complex. Devs who only ever knew the mono-threaded JavaScript ecosystem and suffered from heavy leftpadism. And those very same surveillance/ad complex bros are now generating countless crap with LLMs, in languages they don't understand. They're just mediocre. That's what happened: the veneration of mediocrity.

keithnz

To talk abstractly about all software, you'd also need to analyze all the software that stays working. This article is just cherry picking a few recent incidents and try's to make some conclusions without really looking at causes and whether that can be generalized to "Software".

doug_durham

This narrative isn't consistent with reality. Cloud computing is much older than 10 years ago. We were doing production grade cloud computing over 16 years ago. The OSS revolution dates to the late 90's. The author seems to think that SWE was in a golden age 10 years ago. Sorry, but it's about the same. We have the same problems in SWE as we had 40 years ago. The scale and ambition of the problems we are solving have increased as compute capabilities have increased.

CharlieDigital

It doesn't have to be this way. AI is great at following patterns. AI is great at writing tests. AI is great at doing work that humans would consider drudgery and then applying "enough" judgement to it. The problem is that most software teams and most engineers are very, very bad at the thing that AI needs to succeed: technical writing. Documenting the patterns, the rules, the heuristics, the SOP, the judgement; developers could barely be bothered to write comments before (which in the AI age are free, infrastructure-less memory that is always "line-of-sight" for the agent and can be used to enrich with keywords). It's not that AI is the problem, it's that we humans are good at resolving these constraints in the real world at a human pace. AI makes that pace impossible for a human to keep up as a trailing activity. Rather teams need to invest more in technical writing and producing the corpus that AI relies on to learn the operational constraints and rules.

titzer

Not everyone is using AI to make shittier software. Most big important software projects I know of are frantically fixing an avalanche of AI-found bugs. Important software is getting more robust and resistant to attacks, while still adding functionality. There's a lot of slop, and it will almost certainly grow quicker than non-slop, but there is a big opportunity for those of us who still work at the bottom to make the pillars more solid, and take on re-engineering projects that would be infeasible otherwise. Case in point: Before Claude a port of my compiler to a new architecture might take anywhere from months to a year. After AI, now that there are multiple backends as examples and an even larger--massive even--test suite, this can take days or less. It took Claude about 3 days to come up with an arm64 backend for Virgil (around April time frame). Then in August it took just 3 hours to build a complete RiscV backend. Both of them pass the entire test suite. And they shook out other bugs elsewhere. Overall, more targets, and more robust. Things can have rocket boosters, you just gotta know where to place them.

Semantic search powered by Rivestack pgvector
13,334 stories · 126,150 chunks indexed