Qwen3.8 27B scores 52 on Artificial Analysis

anana_

For more context, this puts it on par with models like GLM 5.2 and GPT 5.6 Luna, which are far larger

apitman

Very interesting. I was not expecting anything close to this.

sottol

A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted? What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?

colingauvin

It's 7th (!!!) overall on the agentic index, above Terra.

beltsazar

As a comparison, Qwen3.6 27B scores 38, which was the highest in its small model category (4B–40B). Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B). Sources: - https://artificialanalysis.ai/models/open-source/small - https://artificialanalysis.ai/models/open-source/medium - https://artificialanalysis.ai/models/open-source/large

johnnyApplePRNG

Unbelievable. Bravo Qwen team.

bertili

I can't shake this the existential feeling that this compact series of 27G bytes represent something profound and universal.

cardboard9926

Where's GLM 5.3 score?

sp1982

Perhaps model size and reasoning length trade off to some extent, similar to CPU vs. RAM. A smaller model with a longer reasoning trace has more intermediate structure to latch onto and build on.

prakashbuilds

Interesting to see where local models are going to be in the coming days. I am already starting to believe open source models are the way to go in the coming days. With Qwen 3.8 Max, Kimi K3 etx already delivering at part perf with frontier models, the future is going to be exciting.

matheusmoreira

It tied with Luna/max. Simply incredible.

JV00

Why is it not included in the Pareto line intelligence/cost chart?

padolsey

The smaller these frontier-nearing models get, the more I'm reminded of https://en.wikipedia.org/wiki/Lottery_ticket_hypothesis

f311a

Why is it so small, but expensive? Open Router Input /M $0.45 Output /M $3.20 Cache read /M $0.05 Throughput 27 tps It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?

manofmanysmiles

Imagine this, and sucesor models on Cerebras or other silicon...

kmike84

I have an internal automated benchmark, which roughly follows my workflow, and I've been testing various models on it, local and cloud. Qwen 3.8 27B did awesome. Its understanding is correct, research is better than e.g. glm's (and I like glm), and implementation is good and careful. Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good. --- (benchmark, to get an idea): 1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking. 2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it. 3. Then, implementation. Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up. Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)

K0IN

I used Qwen 3.6 27B extensively (>1B tokens) and DeepSeek V4 Flash (the older one also 2B+ tokens). And I just can't fathom that the new 3.8 beats the new DeepSeek V4 Flash (which, in my eyes, is one of the best everyday coding models). What an insane release, and convenient size to use every day/locally. but i will test this model extensivly.

armcat

So it's effectively on-par with GLM 5.2 and GPT 5.6 Luna?

IronWolve

Anyone try the 9B/2B distills yet? Wondering how they do for local tools

dethos

I'm impressed with the score. This is a model that runs on a good, but still regular, desktop PC.