What is the best harness for each model?

Edward40 357 points 124 comments November 17, 2026
runta.com · View on Hacker News

Discussion Highlights (20 comments)

guanlan

Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly. The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens. What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.

Majromax

Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps. The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history. I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.

Zigurd

I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.

cronin101

I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)? Can anyone confirm/deny this is the case? If so, it seems a shame not to include this practice into the reporting methodology for full transparency.

jrmg

"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title. The page's title is: "Introducing FrontierHarness Eval v1.1"

ford

To be clear - "Failed" here means "Did not meet all pass criteria on a given benchmark task", not "Experienced a runtime failure"

scubazealous

While this could be better, I wonder how much time the average developer spends on runs that fail? How much time do students spend on runs that fail?

seizethecheese

The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers. It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes. Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs". Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/ . The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.

andai

>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience. I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task. And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)

shevy-java

Well, it is a milking system. You have to pay for those tokens anyway.

colinmarc

How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.

bisonbear

These are cool results and I'm glad that tokens are being invested to eval harness capabilities. However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail. I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!) https://www.stet.sh/blog/cost-per-successful-task

fennecfoxy

Yeah, developers sometimes go down the wrong path as well. It's a non deterministic system exploring a huge problem space...

augment_me

This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context. If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.

underdeserver

Wait till you hear what the human success rate used to be.

maerF0x0

So? I'm sure exceedingly high amount of EC2 cpus are idling, or lost to discarded branch predictions etc. We waste whatever is cheap to gain what is dear. (Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)

bob1029

The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.

evalstate

Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/ GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.

WorldPeas

evolving harnesses are also a cool development I've seen recently though I feel they'll have diminishing returns https://www.alphaxiv.org/abs/2609.01437

jedisct1

It would have been nice to see swival.dev in that list. But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.

Semantic search powered by Rivestack pgvector
13,334 stories · 126,150 chunks indexed