AI handles incidents, engineers lose touch with their systems

I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.

I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.

I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.

Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.

Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.

Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.

The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.

Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.

I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.

Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.

This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.

They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.

You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.

You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.

It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.

I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.

If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.

This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.

There is only so much need for taste and judgement, before even that has been incorporated into the models.

Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.

If an RPA team ever gets direct access to a production database in your company, look for a new job.

Despite not having written code for about 6 months due to using Claude etc, I've surprised myself with how much the "muscle memory" of leetcode etc has come back to me.

I say this b/c I think the veterans with years of coding, debugging etc experience can just pick back up the skills they need even if they're a bit rusty.

The junior folks on the other hand, I'm not so sure. Friends of mine that hire straight out of college are saying that a combination of going to college during COVID + LLMs + "cloud is all you need" has resulted in juniors who don't know how to solve problems nor architect basic software.

I guess we are all on some kind of chart where the slope of losing the experienced people who know what's going on and AI getting so good we don't need people are going to intersect at some point. And that intersection may be later than is good for society and engineering overall.

They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time.

I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue.

This is what progress looks like. We used to do a thing and now we don't.

Depending on what your goals as the author are, you may or may not want that.

Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.

Progress usually means adding levels of abstraction so that more can be done with less effort and needing to worry about lower levels.

AI isn’t all that different.

My intuition and instincts have been honed from years of troubleshooting complex systems so I find it refreshing not having to write probe scripts, queries or code under pressure. I can have a theory and test it very quickly and if it’s not the problem, I can quickly try something else. This greatly improves the time it takes to find the real root cause. Once found, I can do more elaborate testing that I might have skipped before due to complexity or time constraints to really have higher certainty.

If there is ever a time where the model does something I don’t understand, I ask it to explain it to me and I increase my knowledge of the system accordingly. It’s always patient and explains things in such depth without any sighs, patronizing, sarcasm, juxtaposition, etc. So refreshing!

I much prefer working with AI than any co-worker and I find I’m far more productive and efficient, especially with bugs and dealing with incidents. There’s no one to convince or argue with, no ego’s to trip over. It’s lovely.

These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given.

In software we still have the agency but we are giving away the understanding. The agency will follow.

Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.

There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.

I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.

We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.

Management in the end cares about the outcome and not how the meat is made.

I propose creating solutions and programs that will help engineers and coders take control of their projects and prevent AI from taking its course. This applies to all areas of life—economics, medicine, and much more.

AI is a wonderful technology that will accelerate our development. But it must remain a tool controlled by humans.

It looks like our brains somehow have to experience the failures during a diagnosis and in gemerak perform this kind of pathfinding by themselves to truly understand the system. I don't know if this has to do with how our brains actually learn.

AI use across my company is variable and I don’t allow any enforcement of mandatory using or not using of it.

I view and tell my employees to view their skills as that of a high-performing athlete. Consistent training of the basics, rapid fire what-if testing, weekly scenarios are all part of the upkeep.

In short, this is what it takes to keep up the skills no matter if you do or don’t use AI and I can imagine the degradation is much faster if you don’t train AND delegate all of your activity to AI.

The comparison with Pilots is bit disjoint one cause the domain varies a lot for every company and product. Coming up with synthetic simulations within every domain is like paying money twice for the same thing, why not let the engineers to handle the real incidents in the first place itself. In fact why not spend some more thought into building better systems.

Imagine a future where humanity understands none of the underlying technology and AI just babysits us on a daily basis. Wall-e.

Programming will devolve into unintelligible anti patterns and will be written in convoluted languages humanity doesn’t understand. But none of this matters because AI will be handling all of it.

It was just making up random stuff about AWS and system resource limitations and when I was asking for the source like from where it got that info, it was like, “I’m sorry I prematurely concluded that without checking sources”.

I would never trust an Agent to resolve incidents ever

One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.

It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...

Even before layoffs many teams just maintained things org has long lost coherent knowledge of

After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn

We do not yet have the next generation systems that will manage AI creation and maintenance of systems.

Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time.

The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation.

The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing.

We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants.

I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then.

We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this.

CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)

We’re entering a new phase of software development, and with every phase there are new challenges.

Some questions the author might want to first answer: 1. If AI is constantly reacting to smaller incidents, is it surfacing larger issues in your codebase and architecture? 2. What kind of new telemetry do we have to build? 3. How do we build new deployment systems that help us validate fixes without requiring hours to go through traditional CI/CD systems

I’m old enough to remember the days when engineers would monkey patch their code on live servers with additional logging and metrics to learn about failure modes during incidents.

Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation.

What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."

An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.

Why does it not apply?

No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.

I feel like AI could easily do the same for servers and various software, BUT you gotta ground it in actual data so it can't make the shit up and just spew out garbage (if you tell it to come up with a potential answer it will even if it doesn't have the proper data to be "sure" of it).

Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.