Back to Home11/18/2025, 5:33:52 AM

Ask HN: Do you A/B test your LLM prompts?

rjfc

1 points

0 comments

Mood

thoughtful

Sentiment

neutral

Category

tech

Key topics

LLM

A/B testing

prompt engineering

dev tools

I'm exploring a dev tool idea that helps you A/B test your prompts, but I'm not sure if there's a need for it. You'd be able to write and version your prompts in a web UI, then A/B test them and see results with metrics you define.

So for example, with a bot that writes cold outbound emails, you can verify whether v1 or v2 of your system prompt results in a better reply rate.

Does anybody currently do something like this or want something like this?

The author is exploring a dev tool idea for A/B testing LLM prompts and is seeking feedback on its potential need and usefulness. The discussion is currently non-existent, but the topic is relevant to LLM and prompt engineering.

Snapshot generated from the HN discussion

Discussion Activity

No activity data yet

We're still syncing comments from Hacker News.

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (0 comments)

Discussion hasn't started yet.

ID: 45961727Type: storyLast synced: 11/18/2025, 5:35:39 AM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

View on HN