DeepSeek published DeepInfer on GitHub on 28 October, the inference engine that serves DeepSeek-V4.1-Flash. No new model shipped with it. What shipped is the machinery that turns a trained model into a service.
The company also disclosed the scale it is built for: a single cluster handling roughly 155 trillion tokens a day.
At that volume, waste compounds
155 trillion tokens a day means GPU time is the dominant cost line. Any resource wasted inside a single round of computation is multiplied by every round, through every hour, on every card.
The specific waste DeepInfer targets is idle time between rounds. In a conventional decode loop, the CPU selects requests, assembles a batch and prepares KV cache metadata, then the GPU runs the forward pass and sampling, then the CPU reads results, updates sequence lengths and request states, and only then does the next round begin.
As kernels such as DeepGEMM and FlashMLA get faster, the GPU’s share of each round shrinks and the CPU’s scheduling time becomes a larger fraction of the total. During decode, where each round moves few tokens, the handoff happens constantly.
Running one round ahead
DeepInfer’s Zero-Overhead Batch Scheduler reorders the loop. While the GPU is running round N, the CPU prepares round N+1 and uses the gap to process sampling results from round N-1. Some work is submitted to the GPU queue early, with dependencies held by CUDA streams and events.
The hard part is not the pipelining, it is the bookkeeping. If the CPU schedules ahead, it does not yet know how many speculative tokens the previous round actually accepted, so query length, key length and rotary position embeddings can all be off. DeepInfer tracks tokens that have been dispatched but not yet committed, folds that in-flight volume into capacity checks, pre-reserves KV cache space, and remaps each round’s request order against the previous round’s saved execution state. Results land on the GPU through a device-side kernel that corrects the lengths.
Memory has the same problem in reverse. In a synchronous loop, a finished request’s metadata can be freed immediately. In an asynchronous pipeline the GPU may still hold a reference. DeepInfer queues those releases and clears them at a later synchronisation point, confirming completion through CUDA events before dropping the reference.
The part that decides cost at scale
DeepSeek-V4.1-Flash runs prefill at expert parallelism of 2 to 8 and decode at 64 to 128. That width spreads the cost of reading MoE weights across many cards, but every card in a communication group has to execute the same step type, or collective communication mismatches.
DeepInfer coordinates this with in-group voting between two modes. One requires every participating rank to be ready before switching to prefill, which avoids idle cards. The other lets a subset of ranks trigger a group-level prefill so small batches do not sit in the queue. Because a queued request does not guarantee a rank has cache capacity for it, the system runs a second confirmation pass and, if only some ranks can schedule, has the rest execute an empty prefill step to keep the communication order intact.
Expert load is the other lever. Routing sends more tokens to hot experts, so a few GPUs do more work and the whole round waits on the slowest. DeepInfer keeps redundant expert replicas on the GPU and redistributes them from runtime statistics, using a 1,000-step measurement window and rebalancing every 10,000 steps. Measured slow-rank load came out at about 1.090 times the average.
What is still closed, and why the hardware list matters
The release covers offline inference. The online serving pieces, including KV cache storage and cross-replica load balancing, are not fully public, which is a deliberate line: the published code shows how DeepSeek arranges computation, communication and scheduling without handing over the parts that make its service economics work.
The feature list also names supported hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. For anyone tracking the export-control argument, that is the sentence to read twice. An engine tuned for both means DeepSeek can scale its serving on domestic silicon, which turns the constraint from whether the chips work into how many can be built.
Editor’s note: This is an adapted translation of the original Leiphone report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://www.leiphone.com/category/ai/HecXToP0EHbPjPd0.html.