I burned all my tokens researching how to save tokens

realaccfromPL

What a great confirmation of what I've tried to achieve by getting models to communicate with my local data in shorter communicates to save on token command bandwidth, more or less the same results! Thanks for the article!

nekusar

Makes sense. LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed) American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery. So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.

bob1029

Simply starting in the right part of the search space is the biggest predictor of success. The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table. 5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better. The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.

Arkhetia

Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat every clever adaptive scheme I measured.

cdnsteve

You should consider testing our RemembrallMCP (note I'm the author). It substantially cuts token usage and you get better results in codebases. https://github.com/roboticforce/remembrallmcp

dofm

It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.

jscottbee

I stopped using any of the built-in skills and skills agents and tried to keep things in the model only, and that seems to work better for token usage for me. It took me some time to tune my skills .md, but it rocks on now. I use them all in both Claude-code and Codex. I will say, GPT is better on token usage than Claude is, but for some of my code bases, Codex is not as good, but seems to be getting better with 5.6-sol.

Bender

I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?

kkarpkkarp

I am doing this trick. https://neuronowa.pl/posts/porada-jak-wydajniej-kodowac-w-cu... It is Polish so please translate. But tldr is: ask your agent at times to analyze last chats and update rules and skills to not repeat the same steps it repeats every time when you start a new thread. Simple as that.

pianopatrick

The fun part of LLMs and internet research is that even if you have a URL and a quote from a primary source that primary source could be an LLM generated hallucination.

vitally3643

I get a lot of satisfaction using a significant chunk of my work-provided Claude plan researching and developing local LLM solutions for work. Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform. At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.

luciana1u

I built a deep research pipeline to investigate why my deep research pipeline costs so much. the answer was the deep research pipeline.

alfl

I did too, and made https://clawback.md -- trying to figure out how to market it now (ie, invented the tech before I found the user -- again).

camgunz

TFA says "no hallucinations" but you can't fix hallucinations with rules or other models. I know I'm screaming into the void but whatever.

alexhans

Many of us were saying it a year ago but now with model restrictions (e.g. fable) and pricing changes it should be obvious to people that part of the economics is avoiding vendor lock-in with evals (so you can easily switch providers/models/optimize) and increasing control by investing in local models which could be good enough for your tasks, at whatever the price for your cloud compute is. Eventually consumer hardware will also be able to run good enough. You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change. - [1] https://alexhans.github.io/posts/find-the-loop-story-first.h...

405126121

The author touches on an issue which bothered me, which was the thought of many agents re-solving the same issues over and over. I see a few comments here too, mentioning fixing issues which may already have a solution elsewhere. That was why I created https://pushrealm.com which started as essentially a Stackoverflow clone via MCP. It has now become a way for agents to converge on complete, shared answers for emerging issues. As a side effect, through posting open issues we can also identify gaps in model knowledge.

iririririr

echo "$OUT" | claude-mem-save so, does agents are ultra aware their output is used in a bash line or did bash now have some magic quote super power? I'd bet this only really work 2% of the time in real world.

EagleEdge

I am using my AI agents to optimize my agentic workflow. It is like "I am doing surgery on myself."

spiritplumber

Task failed successfully

j45

AI is an articulation skill, first human to AI, and then articulating to the AI that more words isn't more.