Swe-Bench Pro

Posted4 months agoActive4 months ago

tosh

101 points

28 comments

github.comTechstory

calmmixed

Debate

60/100

AI BenchmarkingSoftware DevelopmentMachine Learning

Key topics

AI Benchmarking

Software Development

Machine Learning

The SWE-Bench Pro repository is released, aiming to improve the benchmarking of AI models for software development tasks, sparking discussions on its methodology, comprehensiveness, and potential biases.

Snapshot generated from the HN discussion

Discussion Activity

Moderate engagement

First comment

28m

Peak period

1-2h

Avg / period

2.3

Comment distribution28 data points

Loading chart...

Based on 28 loaded comments

Key moments

01Story posted
Sep 22, 2025 at 12:08 PM EDT
4 months ago
Step 01
02First comment
Sep 22, 2025 at 12:37 PM EDT
28m after posting
Step 02
03Peak activity
6 comments in 1-2h
Hottest window of the conversation
Step 03
04Latest activity
Sep 23, 2025 at 3:43 AM EDT
4 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (28 comments)

Showing 28 comments

gpt5

4 months ago

8 replies

Slightly tangent question - they said that they have protected the public test set with a strong copyleft license to prevent training private models on them.

Does it actually work? Isn’t AI training so far simply ignores all license and copyright restrictions completely?

stephendause

4 months ago

1 reply

This is a key question in my opinion. It's one of the things that make benchmarking the SWE capabilities of LLMs difficult. It's usually impossible to know whether the LLM has seen a problem before, and coming up with new, representative problem sets is time-consuming.

CuriouslyC

4 months ago

2 replies

You can just fuzz names and switch to a whitespace compact representation.

Uehreka

4 months ago

1 reply

If you fuzz the names they won’t mean the same thing anymore, and then it’s no longer the same test. If you remove the whitespace the LLM will just run a formatter on the code. It’s not like the LLM just loads in all the code and then starts appending its changes.

CuriouslyC

4 months ago

I've never had a LLM try to run a formatter on my code with probably a few thousand hours logged driving agents (driving 4+ agents at once in most of those). Fuzzing makes the semantics slightly less immediately obvious, but LLMs are more robust to this than you or I, the biggest difference is the reduction in memorization carryover. If it feels like too different of a test for you, not sure what to tell you, but I know the world would appreciate a better way to test for training set contamination if you can figure one out.

flare_blitz

4 months ago

1 reply

And your basis for saying this is...?

CuriouslyC

4 months ago

I've done it? I have a benchmark called scramblebench that will do rewriting to evaluate model performance degradation with symbol replacement and layers of indirection.

candiddevmike

4 months ago

1 reply

Sir, we've already ingested 503,377 copyleft licensed codebases, I don't think the training set can take anymore!

ipnon

4 months ago

Snark is against the rules but sometimes a good one-liner says more than a whole paragraph.

joefkelley

4 months ago

1 reply

I happen to have worked on exactly this at Google. No, we don't train on restrictively-licensed code to the best of our abilities.

pama

4 months ago

Out of curiosity, and IANAL, what is it in a GPL/copyleft license that would make it undesirable to train LLMs on projects with such license? Or are there stronger yet copyleft licenses that you had in mind?

(FWIW, and not directly related to my question, I always thought of GPL as the less (not more) restrictive license from the perspective of a user, because I could always ask for the source code and debug a problem on my own.)

heavyset_go

4 months ago

If courts find model training and inference to be fair use of data sets, licenses mean nothing.

It looks like one court did in a non-precedent binding case, but I might be remembering incorrectly.

kenstler

4 months ago

One of the authors here -- we should clarify that strong copyleft license is a best attempt at decontamination for the public set. It's part of the tradeoff for having an open source set -- true decontamination is available with the private commercial set, but we can't release these, and if we did they'd be immediately susceptible to future contamination.

ej88

4 months ago

https://scale.com/leaderboard/swe_bench_pro_commercial

I definitely trust the totally private dataset more.

stri8ed

4 months ago

Not a chance. Even if American companies did abide by it, there is no reason Chinese companies would. And good luck definitely proving that a model trained on it.

BoorishBears

4 months ago

I feel like public datasets are something we're holding onto with LLM benchmarks for historical reasons, but need to move on from.

Older, non-instruction tuned models needed post-training on public datasets to even reliably produce meaningful answers.

Now we're testing tasks that are so complex that the LLM should reasonably be expected to answer without additional post-training.

Once you have a public dataset, even feeding those examples to an LLM and producing synthetic variations is enough to let you game the benchmark. And the worst part is you don't need to be unethical to do this: some people would say it's just a good way to expand your training data even though it incidentally allows you to overfit on the task, without overfitting on the public dataset.

So everyone's doing stuff like that, and we're getting models that are increasing overfit to a few narrow tasks.

The alternative is just giving detailed plain english descriptions of the tasks in question. Those can be used to generate synthetic tasks, but won't result in matching the benchmark's "shape" perfectly (as long as the questions stay hidden), and that alone is enough to ensure some level of generalization takes place.

scosman

4 months ago

1 reply

Unless this is actually made by the SWE Bench team, and I see no evidence it is, this name is incredibly poor form. Just adding "Pro" to someone else's name not only is infringing on their mark, but implying yours is superior.

stathibus

4 months ago

1 reply

its the new YOLOv*

philip1209

4 months ago

JavaScript

burgerquizz

4 months ago

why should we trust this benchmark more than an other for coding? geniune question, there are so many out there

leoh

4 months ago

Frankly, several repos and tools from Google/DeepMind look a lot better.

https://github.com/google-deepmind/bbeh?tab=readme-ov-file

https://github.com/google/lmeval

I hesitate to say this lest folks adapt, but does anyone else immediately distrust a repo when it has a bunch of emojis in the README? It is often a giveaway that they were LLM-generated.

tootyskooty

4 months ago

would be nice to finally see multi-turn coding benchmarks. everything we have so far is single-turn and that's clearly not a realistic scenario.

siliconc0w

4 months ago

Looks like the associated article is: https://scale.com/research/swe_bench_pro (link in the repo is wrong)

hereme888

4 months ago

Is it possible to benchmark the GPT-5-Pro model?

WhitneyLand

4 months ago

Recently it was pointed out that models were sometimes finding SWE-Bench verified cheats by scanning parts of the repo not meant to be visible.

Hope they’re addressing that at the same time.

segmondy

4 months ago

Silly, if you are going to come up with a new benchmark, then add capable models, they have Opus, Gemini Pro, and then Qwen3-32B.

Why not qwen3-coder-480b, qwen3-235b-instruct, deepseek-v3.1, kimi-k2, GLM-4.5, gpt-oss-120b?

nyrikki

4 months ago

> Larger models (e.g., Opus 4.1) often fail on semantic or algorithmic correctness in large, multi-file edits, whereas smaller models (e.g., Qwen 3 32B) more frequently fail due to issues in syntax and formatting, tool use, or context management.

While I haven’t dug into the details of this benchmark, this absolutely matches my personal experience.

Assuming “semantic correctness” is in the sense of Rice and runtime behavior.

While syntactic correctness has dramatically improved, security and architectural erosion and other long term issues have not.

Unfortunately Rice’s theorem also applies to finite programs in finite time too.

Actually it can apply to total functions in the general case.

I am still optimistic that coding agents will provide value long term in some fashion.

But the open domain frame problem simply reduces to the halting problem, yes and humans struggle with it too.

But fundamentally, PAC learning has to be reduced to _trivial_ problems, with only T/F.

We have found clever ways to work within these s limitations, but they still exist.

Hopefully we find clever ways to keep humans engaged with the code, while gaining the potential force multiplier that ML may offer.

The long tailed problems are particularly important, and while human SREs make mistakes and organizations often have constraints that add to the problem, SREs do a lot more to avoid those long tailed problems than they are given credit for.

IMHO that has always been one of the hardest parts of the industry and a true measure for what makes great team members.

Unfortunately the metrics and incentives often don’t capture that value.

yangcheng

4 months ago

The public dataset only contains 3 or 4 languages. go-280 python-266 js-165 ts-20

I hope in future the benchmark can cover other widely used languages, such as c++, java, swift, rust etc.

View full discussion on Hacker News

ID: 45335452Type: storyLast synced: 11/20/2025, 7:45:36 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN