Why harnesses matter for energy (and token) efficiency

ccgibson

I'm the CEO of Neuralwatt, an inference cloud that bills on measured GPU energy. I wrote the post. The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size. Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth. Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt. Happy to discuss the methodology.

randysalami

This article mainly argues around harnesses but a breakthrough for me was understanding why harnesses. Without a harness and only a raw LLM, exploring and modifying codebases is not feasible. It’s just text in, text out, and has no way of handling the context of the codebase (if large enough). Plug an LLM into a harness, and all of the sudden it has an exosuit that lets it funnel its thinking into discrete actions. It can now use strategies to manage that large context in pieces.

zby

I don't know - all of these cache hit rate advocacy is kind of goodhearting. Cache hit rates are less important than the accuracy of LLM answers and people seem to forget about it. For example I've seen articles how sub-agents poison the cache. Sure you can fill up your context with irrelevant data to have high cache hit rate - but that degrades the LLM work - sub-agents are about providing the LLM with a clean context - just the information needed to do its job, no distraction. All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.

hedgehog

The thing that surprises me is that any of the popular harnesses break cache this way. Is it just an artifact of too much LLM code and not enough tests?