Running a 28.9M parameter LLM on an $8 microcontroller

cr125rider

9.7 tokens/sec actually seems like a lot! That’s fun!

althea_tx

This is a really cool project. Thanks for sharing!

rao-v

This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!

chrishynes

Why can't this scale to run much larger models on CPU backed by flash with good access patterns?

smy20011

Super cool, thank you for sharing!

spacedoutman

>esp32-s3 This microcontroller is a beast, currently using it to do dev work on a pi4. Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise

titzer

It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!

ReactiveJelly

Finally, an LLM that can run on my i5

NooneAtAll3

While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights