What is the best harness for each model?
Edward40
357 points
124 comments
November 17, 2026
Related Discussions
Found 5 related stories in 129.0ms across 13,334 title embeddings via pgvector HNSW
- Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x shiqimei · 72 pts · September 02, 2026 · 68% similar
- For Computer Use, the harness matters as much as the model MiguelG719 · 29 pts · September 24, 2026 · 62% similar
- What Is a Harness? tosh · 27 pts · August 23, 2026 · 60% similar
- The Harness Is the Thing sfryxell · 81 pts · August 26, 2026 · 58% similar
- I tested 10 model/harness combinations on the same Three.js task alvins82 · 31 pts · September 08, 2026 · 58% similar
Discussion Highlights (20 comments)
guanlan
Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly. The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens. What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
Majromax
Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps. The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history. I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
Zigurd
I find myself pulled in two directions by the performance of coding agents: First, keeping them on a short leash seems sensible. I don't get failed results very often. When I first started using coding agents, this seemed not just sensible but required in order to not deal with a lot of hallucinated weirdness. Secondly though, it's pretty cool to vaguely wave your hands in a certain direction and get both user interface and business logic that's objectively nicer than I would've gotten from the effort I would've put into manually coding. Whenever I cautiously use a coding agent, I wonder what I am missing by not letting it cook.
cronin101
I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)? Can anyone confirm/deny this is the case? If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
jrmg
"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title. The page's title is: "Introducing FrontierHarness Eval v1.1"
ford
To be clear - "Failed" here means "Did not meet all pass criteria on a given benchmark task", not "Experienced a runtime failure"
scubazealous
While this could be better, I wonder how much time the average developer spends on runs that fail? How much time do students spend on runs that fail?
seizethecheese
The headline here is confusing. They ran a bunch of harness and agent combinations on terminal bench tasks, and 64% of the spending was on tasks that didn’t pass terminal bench verifiers. It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes. Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs". Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/ . The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
andai
>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience. I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task. And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
shevy-java
Well, it is a milking system. You have to pay for those tokens anyway.
colinmarc
How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.
bisonbear
These are cool results and I'm glad that tokens are being invested to eval harness capabilities. However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail. I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!) https://www.stet.sh/blog/cost-per-successful-task
fennecfoxy
Yeah, developers sometimes go down the wrong path as well. It's a non deterministic system exploring a huge problem space...
augment_me
This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context. If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
underdeserver
Wait till you hear what the human success rate used to be.
maerF0x0
So? I'm sure exceedingly high amount of EC2 cpus are idling, or lost to discarded branch predictions etc. We waste whatever is cheap to gain what is dear. (Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)
bob1029
The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.
evalstate
Just use a harness that has been properly integrated and tested with each model, with published traces and comparisons (author of) https://fast-agent.ai/benchmarks/ GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
WorldPeas
evolving harnesses are also a cool development I've seen recently though I feel they'll have diminishing returns https://www.alphaxiv.org/abs/2609.01437
jedisct1
It would have been nice to see swival.dev in that list. But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.