Beliefs That Are True for Regular Software but False When Applied to AI

And more often than not the issue is a local configuration issue, bad test data, a misunderstanding of what the code is supposed to do, not being aware of some alternate execution path or other pre/post processing that is running, some known issue that we've decided not to fix for some reason, etc. (And of course sometimes we do actually discover a completely new bug, but it's rare).

To be clear, there are certainly code quality issues present that make modifications to the code costly and risky. But the code itself is quite reliable, as most bugs have been found and fixed over the years. And a lot of the messy bits in the code are actually important usability enhancements that get bolted on after the fact in response to real-world user feedback.

Reality is management is often misaligned with proper software engineering craftsmanship at every org I've worked at except one, and that was because the top director who oversaw all of us was also a developer and he let our team lead direct us whichever way he wanted us to.

It's why I don't play the new trackmania.

Windows is only stabilizing because it's basically dead. All the activity is in the higher layers, where they are racking their brains on how to enshittify the experience, and extract value out of the remaining users.

Last year I assembled a retro PC (Pentium 2, Riva TNT 2 Ultra, Sound Blaster AWE64 Gold) running Windows 98 to remember my childhood, and it is more stable than what I remembered, but still way worse than modern systems. There are plenty of games that will refuse to work for whatever reason, or that will crash the whole OS, specially when existing, and require a hard reboot.

Oh and at least in the '90s you could already ship patches, we used to get them with the floppies and later CDs provided by magazines.

As I mess around with these old machines for fun in my free time, I encounter these kinds of crashes pretty dang often. Its hard to tell if its just the old hardware is broken in odd ways or not so I can't fully say its the old software, but things are definitely pretty unreliable on old desktop Windows running old desktop Windows apps.

(Has anyone tried an LLM on an in-basket test? [1] That's a basic test for managers.)

[1] https://en.wikipedia.org/wiki/In-basket_test

Perhaps because most of the smartest people I know are regularly irrational or impulsive :)

The cases I'm thinking about are things that could be solved in a few minutes by someone who knows what the issue is and how to use the tools involved. I spent around two days trying to debug one recent issue. A coworker who was a bit more familiar with the library involved figured it out in an hour or two. But in parallel with that, we also asked the library's author, who immediately identified the issue.

I'm not sure how to fit a problem like that into this "duration of human time needed to complete a task" framework.

> Once a bug is fixed, it won’t come back again

Regressions are extremely common, which is why regression tests are so important. It is definitely not uncommon for bugs that were fixed once to come back again. This statement doesn't "mostly hold".

> If you give specifications beforehand, you can get software that meets those specifications

In theory, maybe, but in practice its messy. It depends on your acceptance testing. It depends on whether stakeholders change their mind during implementation (not uncommon). It depends on whether the specification was complete and nothing new is learned during development (almost never the case). Providing a specification in advance does not necessarily mean what you get out the other end meets that specification, unless its a relatively small, trivial, non-changing piece and the stakeholders have the discipline to not try change things part-way through. I mean, sure, it may be true that if you throw a spec over the wall, then you will get something thrown back that meets the spec, but the real world isn't so simple.

The fact is, we kind of know how to prevent problems in AI systems:

- Good benchmarks. People said several times that LLMs display erratic behavior that could be prevented. Instead of adjusting the benchmarks (which would slow down development), they ignored the issues.

- Accountability frameworks. Who is responsible when an AI fails? How the company responsible for the model is going to make up for it? That was a demand from the very beginning. There are no such accountability systems in place. It's a clown fiesta.

- Slowing down. If you have a buggy product, you don't scale it. First, you try to understand the problem. This was the opposite of what happened, and at the time, they lied that scaling would solve the issues (when in fact many people knew for a fact that scaling wouldn't solve shit).

Yes, it's kind of different. But it's a different we already know. Stop pushing this idea that this stuff is completely new.

There are people (definitely not me) who buy 100% of anything AI they read. To that beginner enthusiastic audience, your text looks like "regular software is old and unintuitive, AI is better because you can just vibe", and the text reinforces that sentiment.