WebLLM: high-performance in-browser LLM inference engine

refulgentis

Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.

TekMol

This seems to be the demo: https://chat.webllm.ai/ I am getting: WebGPUNotAvailableError: WebGPU is not supported in your current environment, but it is necessary to run the WebLLM engine. On both, FireFox and Chromium on Linux.

conceptme

Bake me a cake responds with > Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.

adastra22

A WebX technology that actually involves browsers!

init0

You might like webml-kit https://npm.im/webml-kit

MarioMan

I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.

mandeepj

It is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.