Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp

thehamkercat

> 11.08× faster and generated tokens 16.36× faster than the same workload in the same stock VM. So this was the comparison, for me the title was a bit confusing

azinman2

I don’t understand what Apple 1-9 are. At first I thought it was M series chips but there is no M9 (yet)

simonw

It looks to me like this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM. The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.

engzaanin

That makes sense. The title initially sounded like a general llama.cpp speedup on Apple Silicon, but if the improvement comes from fixing kernel selection inside Virtualization.framework VMs, that distinction is pretty important.

shay_ker

I recall there was another YC startup that was working on Mac-specific ML optimizations for local inference (and perhaps fine-tuning). I wonder if their work is related?

woadwarrior01

The Claudish in the blogpost makes it really hard to ready. Also, TinyLlama 1.1B lol.

luciana1u

my whole setup is buy more RAM, run it on CPU, and tell myself the GPU is just a personality trait I'm working on.

aeriose

What I don't get, which this article doesn't talk about, why would Apple’s Virtualization.framework expose a lesser Metal profile instead of reporting all capabilities supported by the host GPU?

gigatexal

All this work to get the Mac to be a platform useful for AI is being done despite Apple's efforts. They're famously pissed at Nvidia since the Nvidia + Intel Macs due to heat and other issues. But then the OS is a bit closed off and they move slow and are more focused on milking iOS and the App Store and services for money BUT the PA-Semi purchase and Apple Silicon and everything following it has made the hardware just so amazing and useful that despite all that people build for it. I love the platform. I'm happy to see people building on it. AND! if we ever get an M7 chip with the rumored 1.5TB of available ram all this work will not have been in vain. You think the ai acceleration is nice in the M5 wait till M7 and M8.

w10-1

Related: does anyone have a basis for guessing whether the Neural Accelerators found in M5 Pro+ (accessed by Metal 4) will make their way into the M6 base processors?

wyzer

I see M1 Ultra host mentioned, are there any M1 pro or M3 pro results? Has anyne tried?