Honda: 2 Years of ML vs 1 Month of Prompting - Heres What We Learned

LLMs still beat a clarifier, because they're able to extract more signals than a text embedding.

It's very difficult to beat an LLM + prompt in terms of semantic extraction.

In my current role this seems like a very interesting approach to keep up with pop culture references and internet speak that can change as quickly as it takes the small ML team I work with to train or re-train a model. The limit is not a tech limitation, it’s a person-hours and data labeling problem like this one.

Given I have some people on my team that like to explore this area I’m going to see if I can run a similar case study to this one to see if it’s actually a fit.

Edit: At the risk of being self deprecating and reductive: I’d say a lot of products I’ve worked on are profitable/meaningful versions of Silicon Valley’s Hot Dog/Not Hot Dog.

But I think it's still an interesting result, because related and similar tasks are everywhere in our modern world, and they tend to have high importance in both business and the public sector, and the older generation of machine learning techniques for handling these tasks we're both sophisticated and to the point where very capable and experienced practitioners might need an R&D cycle just to conclude if the problem was solvable with the available data up to the desired standard.

LLM's represent a tremendous advancement in our ability as a society to deal with these kinds of tasks. So yes, it's a limited range of specific tasks, and success is found within a limited set of criteria, but it's a very important tasks and enough of those criteria are met in practice that I think this result is interesting and generalizable.

That doesn't mean we should fire all of our data scientists and let junior programmers just have at it with the LLM, because you still need to put together a good day to say, makes sense of the results, and iterate intelligently, especially given that these models tend to be expensive to run. It does however mean that existing data teams must be open to adopting LLMs instead of traditional model fitting.

Given that it was inside a 9-step text preprocessing pipeline, it would be surprising if the AI had that much autonomy.

Does it make text more clear? How exactly? Does the German language is more descriptive? Does it somehow expands context?

So many questions in this fun fact.

Looks like they were limited by AWS Bedrock options.

But no, they want to pay $0.1 per request to recognize if a photo has a person in it by asking a multimodal LLM deployed across 8x GPUs, for some reason, instead of just spending some hours with CLIP and run it effectively even on CPU.

I won’t lie that I’ve been unreasonably annoyed that I have to use a lot more compute than I need, for no other reason than an LLM API exists and it’s good enough in a relatively small throughput application.

> That’s not a marginal improvement; it’s a different way of building classifiers.

They've replaced an em-dash with a semi-colon.

I thought maybe they did it on purpose at first, like a cheeky but too subtle joke about LLM usage, but when it happened twice near the end of the post I just acknowledged, yeah, they did the thing. At least it was at the end or I might have stopped reading way earlier.

I think we're getting into reverse slop discrimination territory now. LLMs have been trained on so much of what we consider "good writing", that actual good writing is now attributed by default to LLMs.

This has been the bottleneck in every ML (not just text/LLM) project I’ve been part of.

Not finding the right AI engineers. Not getting the MLops textbook perfect using the latest trends.

It’s the collecting enough high quality data and getting it properly annotated and verified. Then doing proper evals with humans in the loop to get it right.

People who only know these projects through headlines and podcasts really don’t like to accept this idea. Everyone wants synthetic data with LLMs doing the annotations and evals because they’ve been sold this idea that the AI will do everything for you, you just need to use it right. Then layer on top of that the idea that the LLMs can also write the code for you and it’s a mess when you have to deal with people who only gain their AI knowledge through headlines, LinkedIn posts, and podcasts.

1. Data collection technique.

2. Data annotation(labelling).

3. Classfier can learn on your "good" negatives — quantitaively depending on the machine residuals/margin/contrastive/triplet losses — i.e. learn the difference between a negative and positive for a classifier at train time and the optimization minima is higher than at test time.

4. Calibration/Reranking and other Post Processing.

My guess is that they hit a sweet spot with the first 3 techniques.

An overwhelming amount of software projects fail, AI just helps them get there faster.

Perhaps I could say, it isn't just generated--it is also hallucinated!

At the same time, as a nonnative speaker of English, this is literally how we were taught to write eye-catching articles and phrases. :P

A lot of formulaic writing is what we were taught to do, especially with more formal things. (This is more of a sidenote to this example)

So in a hunt for LLMs, we also get hit.

overall I think things have gotten better. I noticed maybe 3 years before chatGPT hit the scene that I would frequent on a page that definitely didn't seem written by a native English speaker. The writing was just weird. I see less of that former style now.

Probably the biggest new trend I notice is this very prominent "Conclusion" block that seems to show up now.

Honestly I'd love to see some data on it. I suspect a lot of "that's LLM slop" isn't and others isn't noticed and lots of LLM tropes were rife within online content long before LLMs but we're now hypersensitive to certain things since they're overused by LLMs.

https://arxiv.org/abs/2510.15061

... Wait a minute!

(Even ironically sometimes observed in cases when the writing is disparaging of AI and the use of AI).

If the subject matter is AI, you should instantly pay attention and look for the signs it was AI assisted or generated outright.

Even if it took $10 to run everything to handle each request, that’s far cheaper than even a minimum wage employee when you consider all of the employment overhead.

[1] specifically https://www.warrantyweek.com/archive/ww20230817.html claims the expectation value of warranty claims for a car is around $650.

Warranty data flows up from the technicians - good luck getting any auto technician to properly tag data. Their job is to fix a specific customer’s problem, not identify systematic issues.

There’s a million things that make the data inherently messy. For example, a technician might replace 5 parts before they finally identify the root cause.

Therefore, you need some sort of department to sit between millions of raw claims and engineering. I would be curious what kind of alternative you have in mind?

Their AI implementations are also awful. Just sample 100 contacts with someone who actually understands the business and see their reaction.