The real prices of frontier models

ignoramous

Tokenzier aside, a report shared on reddit found that the GPT 5.6 (edit: 5.5) series are incredibly thrifty with CoTs, resulting in cheaper bills than GLM 5.2 (let alone Opus/Fable): https://www.reddit.com/r/ZaiGLM/s/rUoG5adkPh Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.

SwellJoe

The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer

lolinder

This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks. Other traits where models differ that have an even greater impact on your total spend: * How much context do they load in to solve a given task? * How long do they spend thinking to get equivalent results? * How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out? * Etc. Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.

semiquaver

Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?

sscaryterry

My take from this is that Anthropic is screwing us again. I hope AMD shows them up again.

Tiberium

Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests: - A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M - A ~30kloc TypeScript codebase: GPT 260K, Claude 437K In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).

diwank

this is surprisingly high delta. to make matters worse, reasoning tokens account for the majority of tokens and they are completely opaque so it's hard to tell how much of that is prose or code

f311a

The real price is how often a model uses subagents that scan your whole repository multiplied by thinking level.

jnwatson

The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.

ricardobeat

I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.

j45

An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation. This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.

drob518

Yes, and verbosity (thinking) is a huge factor.

daft_pink

Honestly, I find performant pricing where they test each model on the same task is much more useful than figuring out the tokens or using the input token tax. The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box. Very interesting article though.

ianberdin

We've switched the default model in playcode.io among Opus 4.8, Opus 4.6, Sonnet 4.6, and Sonnet 5. I must admit, Opus 4.8 is quite expensive, and the costs accumulate quickly. Opus 4.6 is about 50% cheaper, while Sonnet 5 is significantly more affordable. According to the data, Sonnet 5 is about 2-3 times cheaper. Fable 5 is unaffordable at all... Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending. I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable. Overall, I really want a benchmark based on real tasks.

yboris

Meta: why are the scrollbars hidden / invisible? Awful UX :(

karma_daemon

The "Honest size of the effect" Honestly, haha

foota

It's a bit unrelated, but I've been wondering if LLM providers are using cache read costs to preserve the illusion of constant output token prices. In reality, the longer your context the more expensive output tokens are, but anthropic and openai both have flat output token pricing. In practice though as a result of cache reads over multiple turns you will end up paying quadratic pricing anyway.

davidw

Where is it getting more tokens in the same source file? Like if you kept minimizing what you're feeding it, what accounts for the difference?

robbie-c

Is it on topic to complain about the various claude-isms in this article? I don't know any actual humans that write titles like "Two floors the rate card hides". I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it? Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.

zaptheimpaler

Wow this is great, it explains what I've seen. My Codex sub seems to last way longer than Claude in real codebases, and Claude eats up a ton of context in the initial read. I thought the harness might be the cause but it seems like tokenization is probably the bulk of it.