Speculative Decoding in vLLM on AMD GPUs

intothemild

Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored. Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro. Really wish this would change.

flufluflufluffy

This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way. Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.

jeanmichelselli

In my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..

myuzio

Very true, I see it for example when my co-workers apply DRY without actually thinking about the complexity it might cause. It objectively makes code smaller, but often comes with complexity cost.

quietraster

nice to see amd getting first-class vllm support. how do the acceptance rates compare to nvidia on the same models?