DeepInfer: High-performance inference framework for DeepSeek models

ggcr

https://x.com/lightseekorg/status/2112470406158561730

schneehertz

A nuclear bomb "Based on this framework, we have achieved high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash. Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (at off-peak pricing)" Source: https://zhuanlan.zhihu.com/p/2095560295930508953

wg0

That's Kami Kaze for US frontier labs. There's no way from here for them towards trillion dollar valuations. That's strategic sabotage to say the least. Ban all open weight models! Cried the VCs.

_aavaa_

Notably missing their kv cache system

jdw64

I think the reason DeepSeek does this is partly to neutralize competitors' technology, but ultimately the core reason might be that data is what matters.

sinisterfeller

Interesting how a comment asking about the reasoning behind releasing this as open source is getting attacked, as if doing a good thing should not ever be questioned. I do wonder of China's motives behind this as well. The government would not allow this open work if it did not benefit them somehow, or perhaps they feel that keeping it closed is not very important right now (maybe because they are somewhat behind the frontier). Comparing a university student Torvalds and state actors open sourcing code is pretty naive...

theplumber

Let’s hope this is just the beginning of bringing the drama queens (Dario and Sam) closer to planet Earth.

xrisk

For comparison, rental prices seem to be ~$2.75/ per H-100-hour [1]. Meanwhile, Deepseek says "38 USD per hour per card”. They haven’t specified which card as far as I can tell. [1] https://www.ft.com/content/a1815bc5-b5d5-47ff-bbbc-e1a689929...

k__

If that's the thing that makes $0.00X cached inputs possible, then this is huge!

GreenWatermelon

Relevant: Deepseek reports its Revenue per GPU per hour at off peak prices (38$) [reddit link] https://www.reddit.com/r/DeepSeek/comments/1xanexq/deepseek_... Also cool (from the translated screenshot since original is in Chineese https://zhuanlan.zhihu.com/p/2095560295930508953 ) > the DeepSeek V4.1 Flash inference system has established three goals > 1-..... 2-..... 3. Supports retaining KVCache for more than 72 hours This is awesome

shafyy

Can somebody explain like I'm five what this exactly is? How are providers like Fireworks or Novita providing inference today? Do they use some other inference engine software? If yes, what is the advantage of the DeepSeek inference engine?

rektlessness

Deepseek released just enough to stir the pot but not enough to cook your own from scratch. Wake me up when we can clone, point it at a dozen H100s we own, and get a working production service. Anthropic and OpenAI will do just fine.

meffmadd

Curious how the performance compares to vLLM or SGLang! They say it was inspired by both so I am wondering what the performance gain is.

lukewarm707

Supported hardware: Huawei Ascend. Deepseek moment.

m00dy

They open source it because they don't need it anymore. Probably they are doing something better with Huawei

VonLuderitz

This month I changed my Claude Max Subscription to DeepSeek API. V4.1 Flash is fast and perfect for everything and refuses to almost nothing. Hope DeepSeek starts month plans soon.

yaoke259

Supporting Huawei Ascend 950 NPUs with open source inference code has huge implications in the global AI hardware market. Imagine if all smaller labs now can build on top of this stack and add increasing support for more compatible hardware , software and models. This could seriously challenge NVidias death grip. Although inference is still only a part of all AI workloads and open sourcing training on Huawei chips would be even bigger.

mekpro

Even though their API price is cheap compared to OpenAI/Anthropic, I still can’t “afford” to use their API because the subscriptions from OAI/Anthropic provide far higher value and peace of mind. I know for sure how much I need to pay each month. I believe most developers fall into this category as well, where $20/$100/$200 from Anthropic/OAI provides more value. This raises the question: why doesn’t DeepSeek provide a subscription service as well? Is it a better move to have a single pricing model like they currently do?

notrealyme123

20 merge requests in the last two hours. And all look somewhat reasonable on first glance. I am happy for not maintaining a relevant open-source repo.

armcat

This is fantastic. I feel the KV cache system is basically a must for any serious inference hosting, both for the provider as well as the consumer. I guess they have this but just haven’t open sourced it yet.