Janas: Run LLMs larger than RAM by streaming experts from SSD

3eb7988a1663

If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62 Which seems sufficient for mortals who are not running gas town, but want a local model.

InsidiousWombat

I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes. Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")

KerrAvon

There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.

purplecats

would this work on a virtual disk, say w s3 or dropbox

kadoban

Why are the models being used so old (in llm years anyway). Why no Qwen 3.8 ?

suprjami

Qwen 3? (Jumanji) What year is it? What a waste of time. Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata

skeledrew

This could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.

zelcon

I think GitHub repos should have a new tag listing the model used to write it along with the token cost. I don’t want to bother reading anything sub-Fable, and this one looks sub-Fable.

jdb9001

Every few weeks, someone stumbles on what llama.cpp has had for years again, and reimplements it. I'm not sure why.

sieve

> Using a GPU has, once, caused a driver reset that took the desktop with it. Only once? This has happened to me so many times with LLM-assisted development. I was writing an OCR engine and pathological shapes would cause OOM that would take the display out. On other occasions, the LLM would try to load 24GB of bf16 weights onto a 16GB GPU without understanding the consequences. Have to encode this in logic: keep some space reserved for the display, and check before allocation. But some errors are bound to slip through. --- Inspired by DwarfStar4 and Strata, I am trying to build my own gemma4-specific inference engine designed to run on gfx1201. It is a lot of work. You cannot simply vibecode one in two days.