AI handles incidents, engineers lose touch with their systems

danielbln

If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice. I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all. If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on. This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields. There is only so much need for taste and judgement, before even that has been incorporated into the models.

_doctor_love

Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM. Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short. There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.

jtfrench

The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started). Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.

bob1029

A natural evolution of engineers losing touch with the customers and users. I'm noticing some of the concern play out regarding AI weakening the capabilities of software people. I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting. I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.

intended

Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.

krtkush

I find the use of AI like quicksand. The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed. Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

ThePhysicist

Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?

everlier

"THIS IS NOT A DRILL"

onion2k

Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong. Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution. If an RPA team ever gets direct access to a production database in your company, look for a new job.

bitlad

We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc. I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes. We recently had a situation, where in with our internal platform and claude we recreated everything in minutes. Management in the end cares about the outcome and not how the meat is made.

hypfer

Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me. Depending on what your goals as the author are, you may or may not want that. Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.

fhub

You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.

iLoveOncall

I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing. No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.

ChiMan

This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.

king_phil

The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not. An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.

devsda

I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system. They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI. You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.

pvtmert

When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust. One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time. It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...

practice9

You rather generously assume engineers are in touch with their systems. Even before layoffs many teams just maintained things org has long lost coherent knowledge of After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn

Traubenfuchs

That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills. Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.

INTPenis

Code too. I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past. Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago. This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.