Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

akie

At which quantization? That's the only thing I want to know. EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.

outfinity

It is 2 bit? anybody tested this? is any good?

intoXbox

This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.

antirez

To put this on perspective, consider that with 128GB of unified RAM it can run even fast : https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.

walrus01

I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded. I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.

armcat

Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance. Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1. [1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855

superkickstart

I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.

spwa4

Anyone else finding these AI generated pages really hard to read?

lastdong

Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!

real_faxenoff

This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.

ctrlaltdileep

hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.

boredatoms

The quantization is probably the most important thing to know and its not clear at all from the dense word salad

yieldcrv

all the pieces are coming into place for the Apple M7 chip, 2028 is going to be great for useful local inference frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute

GypsyKing716

Just because you CAN load it, doesn't mean you SHOULD load it. Sparse models n such ... but I digress.

SillyUsername

Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing. - Bonsai Qwen 3.8 27B - Strata - Pulsar - This... I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers. I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious. So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation. I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional. At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.

throw10920

"2-bit quant" is the "in mice" of "look at how big a model I can fit on this hardware".

dgreensp

How can CPU be fast enough? What does “modchip” refer to?

robotswantdata

Prefill of this approach (prompt processing) is far too slow. The readme is practically unreadable ai slop as well

kennywinker

The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago. I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?

gtirloni

Do people remember humans read README.md files too or are we all just asking for the TL;DR; from AI slop by default?