Amazon's AI boom is creating mess of duplicate tools and data inside the company

Posted by cebert 1 day ago

Counter42Comment18OpenOriginal

Comments

Comment by cowartc 23 hours ago

This is a symptom of the problem. The real issue is that everyone is running off and building their own thing without tying back to a north star and coordinating. I've seen this play out before in a F200. Tooling proliferation resolves itself once everything is driving towards the same goal and owns it. Without that, you're just duplicating symptoms.

Comment by recursivecaveat 18 hours ago

I'm definitely seeing an ongoing train of dashboards, chatbots, and digests. Some of it is definitely self-promotion. I do think though that a lot of people don't realize how much continuous advocacy and support you have to provide for a tool to gain mind share. I've had people message me months after I would have assume that they integrated a tool into their workflow, asking after points of confusion getting started. They love it, super helpful, and then they come back with new bugs you have to triage, new feature requests to support their work. People have limited time and energy, and they will not invest unless you are out pounding the pavement for your tool.

Comment by DonsDiscountGas 22 hours ago

It's worth considering whether that's really a problem. With AI, It's easier to reinvent the wheel 1000 times then get 1000 people together and agree on requirements (which most people would then modify).

Comment by acdha 20 hours ago

There’s some truth to that but I think it’s a huge problem for anything which isn’t isolated to a single team because you’re giving up a shared consensus reality. The article mentions things like copies of specs or datasets becoming stale or missing updates, and I’d also expect problems for many analytic processes — e.g. if we don’t calculate daily active users the same way, we might make conflicting decisions and both be convinced that the other team is wrong.

Comment by cdrnsf 21 hours ago

I'd imagine there's a wide gap in the quality, security and reliability of the two approaches.

Comment by andriy_koval 19 hours ago

The real problem is maintaining, troubleshooting, debugging, extending 1000 wheels instead of one.

Comment by belval 18 hours ago

Bah the other wheels usually die of natural causes if you wait a bit.

I am a big believer in Amazon's "1 > 2 > 0" for "one perfect solution is better than two, but both are much better than no solutions". It's also misunderstanding how Amazon works. If you want to move fast, you can't wait for an SVP 6 levels above you to approve every effort. Instead you build something and then you run it up the chain to have it be adopted at team/org-level.

Comment by lqstuart 20 hours ago

My company has four (4) vibe-coded dashboards to monitor AI tool usage.

We have made no revenue, let alone profit from any AI feature. However, some curiously under qualified people have been hired into new “AI” themed roles with seven or eight figure comp, and we seem to be preparing for major layoffs in the next 30-60 days. Presumably those new roles will be safe.

Comment by CharlieDigital 1 day ago

I have found this at a different scale in our company: agents keep writing the same private static utility methods over and over again without checking for it in existing code.

Sometimes, I'll catch it writing the same logic 2x in the same PR (recent example: conversion of MIME type to extension for images). At our scale, it is still possible to catch this and have these pulled out or use existing ones.

I've been mulling whether microservices make more sense now as isolation boundaries for teams. If a team duplicates a capability internally within that boundary, is it a big deal? Not clear to me.

Comment by appplication 23 hours ago

Let me preface this by saying I have been writing very exacting code for most of my career to a high standard. But with AI generated code, I’m not sure if all the same value-prop exists that were used to with traditional hand-written code.

For example if AI generates 2x of a utility function that does the same thing, yes that is not an ideal, but is also fairly minor in terms of tech debt. I think as long as all behaviors introduced by new code are comprehensively tested, it becomes less significant that there can be some level of code duplication.

It also is something that can be periodically caught and cleaned up fairly easily by an agent tasked to look for it as part of review and/or regular sessions to reduce tech debt.

A lot of this is adapting to new normal for me, and there is some level of discomfort here. But I think: if I were the director of an engineering org and I learned different teams under me had a number of duplicated utility functions (or even competing services in the same niche), would this bother me? Would it be a priority for me to fix? I think I’d prefer it weren’t so, but probably would not rise to the level of needing specific prioritization unless it impacted velocity and/or stability.

Comment by CharlieDigital 20 hours ago

My take is similar.

Majority of the cases, I think this is harmless. In C#, for example, we have agents repeatedly generating switch expressions for file extensions to MIME-type.

This is harmless since there's no business logic.

But we also have some cases where phone number processing gets semi-duplicated. Here, it's a bit more nebulous since it looked like it was isolated, but still had some overlapping logic. What if we change vendors in the future and we need a different format? Have to find all the places it occurs and there's now no single entry point or specific pattern to search for.

Agents themselves may or may not find all the cases since it is using `grep` and doesn't have semantic understanding of the code. What if we ask for it to refactor and its `grep` misses some pattern?

Still uneasy, but yet to feel the pain on this one.

Comment by mikodin 22 hours ago

> For example if AI generates 2x of a utility function that does the same thing, yes that is not an ideal, but is also fairly minor in terms of tech debt. I think as long as all behaviors introduced by new code are comprehensively tested, it becomes less significant that there can be some level of code duplication.

We still run into the same issues that this brings about in the first place, AI or no AI. When requirements change will it update both functions? If it is rewriting them because it didn't see it existed in the first place, probably not. And there will likely be slight variations in function / components names, so it wouldn't be a clean grep to make the changes.

It may not impact velocity or stability in the exact moment, but in 6 months or a year - it likely will, the classic trope of tech debt.

I have no solution for this, it's definitely a tricky balance and one that we've been struggling with human written code since the dawn.

Comment by odux 21 hours ago

Duplicate tools and data within Amazon is as old as Amazon. It is even mentioned in the Leadership Principles or related documents somewhere.

Comment by minimally 1 day ago

http://archive.today/06p42

Comment by LeCompteSftware 1 day ago

Brooks's Law: Adding manpower to a late software project makes it later. https://en.wikipedia.org/wiki/The_Mythical_Man-Month

With the obvious preface of "thoughtlessly adding." Of course it's not a real law, it's a tongue-in-cheek observation about how things have tended to go wrong empirically, and highlights the unique complexity of adding manpower to software vs. physical industry. Regardless, it has been endlessly frustrating for people to push AI/agentic development by highlighting short-term productivity, without making any real attempt to reconcile with these serious long-term technical management problems. Just apply a thick daub of Claude concealer, and ship.

Maybe people are right about the short-term productivity making it worthwhile. I don't know, and you don't either: not enough time has elapsed to falsify anything. But it sure seems like Fred Brooks was correct about the long-term technical management problems: adding Claudes to a late C compiler makes it later.

  The resulting compiler has nearly reached the limits of Opus’s abilities. I tried (hard!) to fix several of the above limitations but wasn’t fully successful. New features and bugfixes frequently broke existing functionality.
https://www.anthropic.com/engineering/building-c-compiler

Comment by dannersy 22 hours ago

Honestly, it isn't particularly good at greenfield stuff. It's a mess all around. Seeing the claims that AI is better at code than humans has been a failure for my direct experience and those I know who work at larger companies. Incident and regression rates are up exponentially and requiring more people to play wack-a-mole.

Comment by socratic_weeb 21 hours ago

Lots of copium on this thread. The reality is that this iteration of AI technology is mostly crap. Moreover, expensive crap. The bubble is popping, sorry.

Comment by andriy_koval 18 hours ago

Its 20x multiplier if used carefully and correctly.