Running Qwen3.8 at 255 tok/s on a single 5090

jkingsman

I struggle with approaching this as I do with many quants, in that there's not really a good measurement of how lobotomized by quantization they are other than "a bit but it's not too bad". Ternary weights immediately make me think "unusably quantized" but I'm presuming that's not the case if someone's standing by this release without massive caveats. I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table. I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way. Frustrating, as a hobbyist.

ranger_danger

How would this compare speedwise to Bonsai 2 on the same GPU? https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf

BizarroLand

The title doesn't apply to the link. The website is not as good as it needs to be. There's no information being shared other than these models exist and I don't want to drill down through 3+ links trying to find what you're saying about them. Who would run a random script on a random page with no other information given? I'm not saying this is bad, but they definitely need to explain what the heck is going on to the point where a 15 year old can understand it, and provide logical and clear breadcrumbs to navigate through their pages. Even a FAQ would help.