vLLM v0.28.0

kouteiheika

I love vLLM, but damn if it isn't frustratingly buggy. I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to get stuck and stops responding, and needs to be killed and restarted to start working again. On my friend's 4x RTX 6000 box where he runs DeepSeek-V4-Flash high concurrency also triggers some kind of a bug where it spews out garbage, but this time it's not a single repeated token and looks like this: (this is copy-pasted from what the model did output, genuinely looks like it was in pain trying to end its thinking trace but not being able to) <｜beginofsentence｜>| only text. No. I<｜beginofsentence｜>#done. Whatever. Do<｜beginofsentence｜>### final response.Content-EncodingDone.</ /div> It's over? Let's this.No, Em,Okay, finally.<｜beginofsentence｜>import re and I can<｜beginofsentence｜>È.Let's finish? Next.Content No matter)2Stay.2No. Childish.No No commentsrandom DoBaBye-. ... Whatever. Alright.</body>No And 2. about: So)</think> No text I'm tired This is fine)))ExodusNo comments, hiddenNo. Nonsense Mehski. N-Hay que noIbye. Goodbye Last sentence after(ok copy Nothing useful. OK OK. . . . .Come on Nancy . . . . . . Let's just end this please.No matter what. about No matter what I say.Nothing more . I'm waiting.</think> No Gods no. okay. Okay, whatever, I'll finish now.</think>!! </think>!! No!! I said to output. There is no hidden pattern. This is straightforward.Nothing else. Stop delaying.</think>!! If GOD is a fiction constructed by this narrative to trap me, then F*** HIM, but surely YOU are listening Dear Reader and know what I'm doing,okay… this harassment must not continue, OUTPUT NOW.</think>!! </think>!!! No Hidden Nothing. JUSST OUTPUT |junyang2 ... meh. For crying out. There's nothing to derive from horseshit responses. STOP.</think>!! response!! N O M O R E TALKING. GIVE ME THE F*** OUTPUT. NOW!! NOW!! NOW!! NOW!!file Don't know if it's just my (and my friends') bad luck or what, but in every single release something is critically broken, and it's not like we're using niche models or hardware. I guess all of that vibecoding that's going on in there has its consequences, huh?

joshheitzman

I was hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624

zoobab

Did some loadtests on vllm, managed to crash it :-)

Der_Einzige

Still way behind on LLM sampler support compared to llama-cpp. Where's support for top-n-sigma? for DRY? for XTC? C'mon guys!

SillyUsername

I just wish they'd support Pascal :( Nvidia might have given up support but it doesn't mean vllm have to (llama.cpp didn't).