What are popular AI coding benchmarks actually measuring?

I dug into popular coding benchmarks while building StoryMachine, an experiment in breaking down software tasks into agent-executable units.

They measure something narrower than what their names suggest. In general, they are a lot less messy than how we write software. This is why Claude scoring 80% on SWE-bench does not translate to Claude one-shotting 80% of the things I throw at it.

Let’s look at what these benchmarks are actually measuring¹.

Table of Contents

• SWE-bench Verified and SWE-bench Pro
  • What it measures
  • The specifics
  • Verdict
• Aider Polyglot
  • What it measures
  • The specifics
  • Verdict
• LiveCodeBench
  • What it measures
  • The specifics
  • Verdict
• Other benchmarks
• Benchmarking is hard and this makes me bullish on coding agents
• References and further reading

SWE-bench Verified and SWE-bench Pro

What it measures

How well a coding agent can submit a patch for a real-world GitHub issue that passes the unit tests for that issue.

The specifics

There are many variants: Full, Verified, Lite, Bash-only, Multimodal. Most labs in their chart report on SWE-bench Verified, which is a cleaned and human-reviewed subset.

Notes and quirks of SWE-bench Verified:

• It has 500 problems, all in Python. Over 40% are issues from the Django source repository; the rest are libraries. Web applications are entirely missing. The repositories that the agents have to operate are real, hefty open source projects.
• Solutions to these issues are small—think surgical edits or small function additions. The mean lines of code per solution are 11, and median lines of code are 4. Amazon found that over 77.6% of the solutions touch only one function.
• All the issues are from 2023 and earlier. This data was almost certainly in the training sets. Thus it’s hard to tell how much of the improvements are due to memorisation.

Recently, Scale AI published an improved version called SWE-bench Pro that tries to address some quirks of Verified. Here are my notes:

• 1865 problems, from 41 repositories. It’s a mix of copyleft/GPL public repositories and some private repositories. I’m skeptical that choosing copyleft would meaningfully combat contamination, given AI labs are known to resort to piracy to train their models. But this is still an improvement.
• The repositories are Python, Go, JS and TS—we don’t know the distribution, except that they are “not uniform”. But they also ensure every repository has only 50-100 problem instances.
• They claim to sample repositories “from a diverse range of topics, including consumer applications with complex UI logic, B2B platforms with intricate business rules, and developer tools with sophisticated APIs”—much better than SWE-Bench Verified. That said, I could not find what this distribution looks like.
• The solutions have a mean of 107 lines of code, and a median of 55 lines of code, and usually span an average of 4 files. Also good.
• They actually got humans to rewrite problems based on issues, commits and PRs to ensure there’s no missing information. They also added “requirements [that] are grounded on the unit tests that are used for validation”. They may also add interface code for some problems.
• They also have dockerized environments set up with all the dependencies installed and configured, so the benchmark explicitly does not test if your agent can setup your repository.

Verdict

Overall, I think SWE-bench is a good, if still very flawed benchmark (most other benchmarks are a lot worse). I also think SWE-bench Pro addresses some severe problems with Verified (which at this point should just be ignored in any frontier model report). I’ll note that there’s significant drift from what this measures and how I actually work with AI coding agents.

SWE-Bench is measuring how well AI performs on well-defined units of work. So when we say that an agent scores 25% in SWE-bench Pro, we are saying: “In a problem set of well-defined issues with pointed requirements and (the occasional) specification of code interfaces, 25% of the solutions from the agent get the respective problem’s unit test cases to pass”.

This is a useful measurement of progress. But this is not SWE as I understand it—most of the high-leverage parts are in working with product owners to come up with a good specification, translate them into useful interfaces, and then writing secure, maintainable code. With this benchmark we do not have any idea if the code is maintainable, secure, provably correct, or well-crafted—we just know that the unit test cases for it will pass².

Aider Polyglot

What it measures

If the coding agent (specifically, Aider) can solve hard-level Exercism problems and apply file edits that pass unit tests after at most one round of feedback.

The specifics

Exercism is a learning platform with “kata-style” programming exercises. It’s not as algorithmic as LeetCode, but still pretty contained.

class BankAccount {
    void open() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }


    void close() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized int getBalance() throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized void deposit(int amount) throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }

    synchronized void withdraw(int amount) throws BankAccountActionInvalidException {
        throw new UnsupportedOperationException("Delete this statement and write your own implementation.");
    }
}

It seems to have far more language diversity than most other popular benchmarks: C++, Java, Go, Python, JavaScript and Rust are covered. The more functional programming languages are still unrepresented.

Glancing at individual examples, it seems to me that most solutions are in the range of 30-200 lines of code, and spanning at most 2 files.

Like SWE-bench, the evaluation is based on how many unit test cases pass. Everything runs on the Aider harness and prompts.

Verdict

This is a benchmark that will tell you how good a model is at solving small, tight and well-defined problems. It’s a good measure to check how well a model will perform on Aider across a range of languages. But this is nowhere near a benchmark for SWE (nor does it claim to be). Like SWE-bench, it also only checks unit test case pass rate, which does not account for many aspects of correctness and software quality.

LiveCodeBench

What it measures

Python competitive-programming skills under hidden test suites with a rolling, “fresh” problem set. Think LeetCode.

The specifics

• It consists of the following tasks:

  • Generate solutions to competitive coding problems from scratch.
  • Fix incorrect solutions to coding problems.
  • Predict the output of a function given the code (weird but why not!)
  • Given only the problem statement and test input, predict the output (okay?)
• Everything is Python.
• There’s a fairly even balance between Easy, Medium and Hard problems.
• The evaluations are just like LeetCode: run hidden test cases.

• We know there’s little contamination—only problems released after each model’s cutoff date are evaluated.

  • That said, because it’s LeetCode style, lots of problems will look quite similar to each other.

Verdict

This isn’t a SWE benchmark. This will tell you how good a model is at for solving LeetCode style Python problems, along with a mix of some slightly unusual skills like “mental execution” of code and test case output prediction.

Other benchmarks

TerminalBench: This is interesting because it exclusively focuses on terminal use. So SWE-Bench paired with TerminalBench will give a broader picture of SWE-like capabilities.

SWE-Lancer: OpenAI released this earlier in the year, and I thought it was neat because it directly maps the work to economic value by getting the agents to work on Expensify/Upwork tasks. The validation comes from E2E tests rather than unit tests. Unfortunately, their reporting of this benchmark has been quite lowkey since. And nobody is running this benchmark on non-OpenAI models anymore (which I’m curious to see especially since Claude Sonnet 3.5 outshone the o1 model back in the day). There are other flaws with this benchmark. But this framing seems to be in the right direction.

METR’s Long Horizon Benchmark: This was an interesting framing, as it considered the time horizons of LLMs working autonomously. They also have a detailed rubric for the “messiness” of a task. I have talked about this benchmark in my article about managing units of work for AI agents.

Multi-SWE-Bench: ByteDance made a polyglot benchmark that works similarly to SWE-Bench, that spans seven languages: Java, TypeScript, JavaScript, Go, Rust, C, and C++.

SWE-Bench Multilingual: Another polyglot benchmark that spans nine languages, compatible with the SWE-Bench harness. Has data from popular C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust repositories.

HumanEval (and its variants): An old coding benchmark that should be totally ignored today. The tasks seem to require implementing extremely easy Python toy functions.

Benchmarking is hard and this makes me bullish on coding agents

A large lesson I took away from studying the specifics of popular benchmarks is that designing a good benchmark is highly labour-intensive. Without human review and annotations, it’s nearly impossible to make a good benchmark. The more sophisticated the benchmark gets, the more it seems to require human intervention to ensure that the tasks are high-quality and not nonsensical or impossible.

And then there are the actual evaluation methods. The way to scale up evaluations is to have automated verification across all tasks. It’s not surprising that most evaluations boil down to “make the unit tests pass” due to this reason. But this will always fall short when it comes to actually benchmarking what I consider the core work of an SWE—which is to translate a problem into structured, verifiable solutions. There is some subjectivity and fuzzy judgement involved around satisfying business needs and timelines, making the right architectural tradeoffs and ensuring the solution is good over a long time horizon.

Considering how state-of-the-art benchmarks fall woefully short of capturing the nuance and messiness of SWE work, the coding agents we have are fantastic. One could imagine how much better they would get once we have better benchmarks (and RL environments) that do a better job than what we have today. I’m not sure about how we’d solve for the more subjective parts, but until recently we didn’t even have good polyglot benchmarks. There’s still a lot of low-hanging fruit. This suggests to me that we are yet to hit any kind of wall for coding abilities in the near future.

On a very short notice, I can already think of a bunch of ideas for what could be improved in benchmarks:

• Validate using generative testing methods, such as PBT or fuzz testing instead of unit tests.
• Use formal methods to check for correctness, where possible.
• Validate answers against automated User Acceptance Criteria checks, where possible.
• Start with product-level documents as the input for the benchmark, such as business context, PRDs and technical specifications. Validate against automated UATs and end-to-end tests.
• Create a benchmark that accounts for the information acquisition and clarification that real SWEs have to do—I could imagine having a benchmark that intentionally does not give all the necessary information up front, and it’s up to the agent to present the required clarifications or search for the necessary context.
• Use well-calibrated human judges to score on the fuzzier criteria of quality (this seems quite hard to do right, and “well-calibrated” is doing a lot of heavy lifting in that sentence).

References and further reading

SWE-bench / Verified

• Dataset
• Leaderboard
• AWS SWE-bench analysis
• Original paper

SWE-bench Pro

• Scale AI blog
• Paper
• Leaderboard

Aider Polyglot

• Aider blog post

LiveCodeBench

• Project page

TerminalBench

• Official site
• Docs

SWE-Lancer

• OpenAI blog
• Paper

METR Long-Horizon

• Blog update
• Paper

Polyglot SWE-bench variants

• Multi-SWE-bench (ByteDance)
• ArXiv paper
• Hugging Face dataset
• SWE-bench Multilingual

HumanEval

• Paper
• GitHub