Astra 6.2 and Sol 6.2 flop at Simple Bench

algoth1 170 points 160 comments November 21, 2026
simple-bench.com · View on Hacker News

Discussion Highlights (20 comments)

algoth1

Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place

bieganski

I tried taking the test, but ultimately stopped at Question 4: There are two sisters, Amy who always speaks mistruths and Sam who always lies. :)

Culonavirus

6.2 is for sure the worst received update OpenAI has ever released. What happened lol. I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.

hoppp

All the new Gemini models perform better on this than Opus 5.5 Is that something that can be noticed during real life use? I really like opus 5.5, It's perfect and I haven't tried the new gemini models.

martinjc

For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.

goodcake3963

Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware

0xbadc0de5

Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2. I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.

xqcgrek2

As the kids say, OpenAI is cooked. Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.

varispeed

I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.

arthur-st

This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.

ricardobeat

Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models? Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.

TrackerFF

Makes me wonder if people are getting different models? So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.

davidmurdoch

I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.

Topfi

Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe. 6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait. There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case. To give some examples: The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back: > message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation. I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me. Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty: > I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list. > Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one. I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great. Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains. I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet. And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between. As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguards public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now). Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, scales with effort levels unlike other Gemini models which play tightrope, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.

loveparade

The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it? Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.

nsingh2

The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics. However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic". All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.

cheema33

Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development.

ModernMech

I just had this very fun interaction with it: Me: Get PR 973 to green Codex: I’ll inspect PR #973 and its failing checks, fix the issues, and run the relevant validation before pushing the changes. Codex: I’ll inspect PR #972 and its failing checks, fix the cause, and push the changes. I’ll use a separate worktree to preserve the uncommitted work in your current checkout. On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.

questionedon

Is a Gemini model worth getting for looking at videos to pull out animations and stuff? because they can look at videos differently/better than other models.

Almondsetat

Honest question: with infrastructure like OpenRouter being a thing... what even is the point of benchmarks? Want to get a feel of how a model might make your job easier? Play with it by switching to it seamlessly and judge for yourself.

Semantic search powered by Rivestack pgvector
13,334 stories · 126,150 chunks indexed