Fixed three bugs that made Qwen3.5-122B a daily driver on Mac Studio

marzukia

I spent three weeks debugging why my Qwen 122B setup on an M3 Ultra was taking 3–5 minutes to generate the first token on follow-up messages (despite having a "warm" context). The root cause wasn't the model, but three specific infrastructure bugs in my serving stack: 1. Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing a full re-compute every turn. 2. Interrupt Path: Streaming replies weren't persisted when the generation was interrupted, causing history divergence. 3. Checkpoint Poison: A background writer created unmatchable checkpoints that crowded out valid ones, triggering aggressive eviction. After fixing these, prefill time dropped from minutes to sub-seconds (53k tokens cached, 33 tokens prefilled). I've open-sourced the fork (qMLX) and a benchmarking tool to verify these numbers. Would love feedback on the hybrid attention caching strategy or any other edge cases I might have missed.

marzukia

The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills. I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.

jval43

Impressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too. One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.

hankbond

I appreciate the amount of detail in the post, I think it's a useful addition to the space. That said, I have to read LLM output all day all the time, and I would implore you to take the time to explore your own voice a bit more. > Two separate things then happened, and it is worth keeping them apart. Is one of those phrases claude spits out nonstop.

tills13

Sorry but I'm not reading an AI slop article no matter how pertinent or interesting the subject is. You want my attention? Earn it. Write it yourself.