Where's the Shovelware? Why AI Coding Claims Don't Add Up

You can only utilize so many people or so much action within a business or idea.

Essentially it's throwing more stupid at a problem.

The reason there are so many layoffs is because of AI creating efficiency. The thing that people don't realize is it's not that one AI robot or GPU is going to replace one human at a one to one ratio. It's going to replace the amount of workload one person can do. Which in turn gets rid of one human employee. It's not that you job isn't taken by AI. It's started. But how much human is needed is where the new supply demand lies and how long the job lasts. There will always be more need for more creative minds. The issue is we are lacking them.

It's incredible how many software engineers I see walking around without jobs. Looking for a job making $100,000 to $200,000 a year. Meanwhile, they have no idea how much money they could save a business. Their creativity was killed by school.

They are relying on somebody to tell them what to do and when nobody's around to tell anybody what to do. They all get stuck. What you are seeing isn't a lack of capability. It's a lack of ability to control direction or create an idea worth following.

I don't, but at least it is somewhat logical. If you truly believe that, you wouldn't necessarily want to hire more developers.

99% of the draw of AI is cutting labor costs, and hiring goes against that.

That said, I don't believe AI productivity claims, just pointing out a factor that could theoretically contribute to your hypothetical.

They didn't claim it was flawless, they had just brought it up because it caused them to question their internal narrative of their own productivity.

> * Their own self-experiment is not exactly statistically sound :)

They didn't claim it was.

> * they assume that all code gets shipped as a product.

The author did not assume this. They assumed that if AI is making developers more productive, that should apply to shovelware developers. That we don't see an increase in shovelware post-AI, makes it very unlikely AI brings an increase in productivity for more complex software.

If AI were really making people 10x more productive, given the number of people who want to make games, you’d expect to see more than a few percent increase year over year.

So amazing that every single stat showed by the author in the article has been flat at best, despite all being based on new development rather than work on existing code-bases.

I wax and wane on this one.

I've had the same feelings, but too often I've peaked behind the curtain, read the docs and got familiar with external dependencies and then realize whatever the LLM responds with paradoxically either wasn't following convention or tried to shoehorn your problem to fit code examples found online, used features inappropriately, took a long roundabout path to do something that can be done simply, etc.

It can feel like magic until you look too closely at it, and I worry that it'll make me complacent with the feeling of understanding without actually taking away an understanding.

sorry, what am I supposed to use on nails?

What is this supposed busy work that can be done in the background unsupervised?

I think it's about time for the AI pushers to be absolutely clear about the actual specific tasks they are having success with. We're all getting a bit tired of the vagueness and hand waving.

> LLMs can get me up to speed on new APIs and libraries far faster than I can myself

To this?

> LLMs can get me up to speed on old APIs and old libraries that are new to me far faster than I can myself

My experience is if the library/API/tool is new then the LLM can't help. But maybe I'm using it wrong.

Weren't the code generators before this even better though? They generated consistent results and were dead quick at doing it.

It didn’t work. I asked a colleague. He had the same problem. Turned out it was using out of date setup instructions for a major tool that has changed post training.

After spending time fixing the problem, I realised (1) it would have been faster to do it myself and (2) I can no longer trust that tool to set anything up - what if it’s doing something else wrong?

Just a random personal anecdote I wanted to throw out. I recently had to build some custom UI with Qt. I hadn't worked with Qt in a decade and barely remembered it. Seems like a perfect use case for AI to get me "up to speed" on the library, right? It's an incredibly well documented library with lots written on it, perfect fodder for an AI to process.

So, I gave it a good description of the widget I was trying to make, what I needed it to look like and how it should be behave, and behold, it spit out the specific widget subclass I should use and how I should be overriding certain methods to customize behavior. Wow, it worked exactly like promised.

So I implemented it like it suggested and was seemingly happy with the results. Went on with working on other parts of the project, dealing with Qt more and more here and there, gaining more and more experience with Qt over time.

A month or two later, after gaining more experience, I looked back at what AI had told me was the right approach on that widget and realized it was completely messed up. It had me subclassing the completely wrong type of widget. I didn't need to override methods and write code to force it to behave the way I wanted. I could instead just make use of a completely different widget that literally supported everything I needed already. I could just call a couple methods on it to customize it. My new version removes 80% of the code that AI had me write, and is simpler, more idiomatic, and actually makes more sense now.

So yeah, now any time I see people write about how "well, it's good for learning new libraries or new languages", I'll have that in the back of my mind. If you don't already know the library/language, you have zero idea whether what the AI teaching you is horrible or not. Whether there's a "right/better" way or not. You think it's helping you out when really you're likely just writing horrible code.

Traditional documentation has always been a challenge for me - figuring out where to start, what syntax conventions are being used, how pieces connect together. Good docs are notoriously hard to write, and even harder to navigate. But now, being able to query an LLM about specific tasks and get direct references to the relevant documentation sections has been a game-changer.

This realization led me to flip my approach entirely. I’ve started heavily documenting my own development process in markdown files - not for humans, but specifically for LLMs to consume. The key insight is thinking of LLMs as amnesiac junior engineers: they’re capable, but they need to be taught what to do every single time. Success comes from getting the right context into them.

Learning how to craft that context is becoming the critical skill.

It’s not about prompting tricks - it’s about building systematic ways to feed LLMs the information they need.

I’ve built up a library of commands and agents for my Claude Code installation inspired by AgentOS (https://github.com/buildermethods/agent-os) to help engineer the required context.

The tool is a stochastic parrot, you need to feed it the right context to get the right answer. It is very good at what it does but you need to use it to its strengths in order to get value from it.

I find people complaining about LLMs often expect vibe coding to be this magic tool that will build the app for you without thinking, which it unfortunately has been sold as, but the reality is more of a fancy prompt based IDE.

This is trivial work that you should have automated after doing it once or twice anyways :/

And if it's not that, then the silly money valuations don't make any sense.

Depending on the amount of code, I see this only as positive? Too often people pull huge libraries for 50 lines of code.

It's difficult because you need team members to be able to work quite independently but knowledge of internal libraries can get so siloed.

Lord, forgive them, they know not what they do.