Mesh LLM: distributed AI computing on iroh

turtleyacht

It sounds like iroh enables distributed compute without having to finangle custom hardware.

jmercouris

I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.

darkpicnic

cocompute.ai is already doing this really well.

SwellJoe

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks. Are we talking 1 token per second for a split model? Less? Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.

darkpicnic

Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?

i386

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

_superposition_

I just wish I had the hardware to try it out!

dwoosley

I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow. What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.

nullc

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.

whatjustin

The real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.

Abishek_Muthian

I'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output. Nicely done OP.

downrightmike

difference between this and Exo?

MattPerry

The first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture. As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.