Why your local LLM feels dumber than it is

jonplackett

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

anotherCodder

most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings

JacobJack

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.

catlifeonmars

> I will make you read the really long unpleasant version with math. This is the version I want to read :) I assume it is unpleasant in spite of the math, not because of it?

InvertedRhodium

I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges. Using oh-my-pi in a prebuilt environment that I let Qwen build too. Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.

IronWolve

sglang, 150+ tok/s on a 5090 in ubuntu 26.04 via wsl. gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, dspark, medium reasoning, 96k context. Using opencode and it built a old fashioned arcade vertical shooter with no issues. Images are ok'ish, just had grok create updated images, and it came out great.

walrus01

Much of this is why I stick to the rule of: a) Don't quantize your KV cache b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.

a11r

Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.

nullpoint420

At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering

fenestella

The section on system prompts and context window management is spot on; most people don't realize how much the default quantization in popular runners degrades logic compared to full FP16. I'd be curious to see if the author has benchmarked the impact of KV cache compression on longer context reasoning, as that usually seems to be where my local Llama 3 setup starts to fall apart.

throwdbaaway

> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ... If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected. > ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’) But this failure mode can still happen. Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better. So, perhaps the lesson here is "don't use vllm at home"?

mrgaro

Any DGX Spark users in this thread? What's your favourite model to run on it?

woadwarrior01

RTN quantization of weights