A code completion task returns 8. Add a few irrelevant tokens to the front of the same prompt and the same model returns 32. ByteDance’s Seed team documented the pattern in late September, and the follow-up question is whether it still reproduces on the shipping model.
The phenomenon is real, the cause is architectural, and it sits inside the compromise every efficient inference stack makes.
Two attention schemes, one weakness
DeepSeek V4 alternates between two mechanisms to hold down the cost of long-context inference. The first, compressed sparse attention, groups eight tokens into a compression window and writes one compressed record every four tokens, then selects whichever records are most relevant to the current query. The second, heavy compression attention, compresses harder at one record per 128 tokens but reads the entire history of compressed records. The first preserves fine-grained retrieval, the second supplies cheap global context.
The Seed paper traced the model’s performance drift to how these mechanisms chunk the key-value cache. The same piece of information can move a few tokens in either direction and land in a different compression slot, changing what is retained and what is retrievable. The researchers call that position the phase. If a model retrieves well from some phases and poorly from others, the answer changes even though the underlying fact does not. They call it phase sensitivity.
The scaled-up test is the convincing part. In a 128K-context experiment where the model had to find a specific item among 16,000 key-value records, the best-performing position group reached 69.9 per cent accuracy and the worst reached 29.7 per cent. The gap of 40.2 points oscillated with a period matching the compression stride.
Ablations narrowed the cause. Removing rotary position embeddings did not remove the weakness. Replacing the learnable compression gate with a uniform average did not remove it either. Changing the stride to 4, 6, 8 and 12 shifted the period of the oscillation accordingly, while a full-attention control model showed no comparable effect. That points at the chunked cache compression rather than at one unlucky completion.
The authors are explicit that 40.2 points is the best-to-worst positional spread of a specific base model in a controlled retrieval experiment, not a measure of how DeepSeek performs in ordinary use.
What happens on the current model
Leiphone rebuilt the paper’s FP8 code completion test and ran it against the shipping V4.1-Flash through the official API. The base model in the original experiment is no longer callable that way, so the comparison is between generations rather than an exact replication. The setup held the code fixed at 983 tokens and varied only the length of an unrelated prefix, across four filler styles and 16 lengths each, moving from 24 to 39 tokens. Adjacent inputs differed by a single token.
In the first round, all 64 fixed-sampling runs returned the correct answer. No flip appeared. Confidence behaved differently. Measuring the probability assigned to 8 and to 32 across 64 runs, 8 remained the top candidate every time, but its probability was not stable: in the docstring group, extending the prefix from 24 to 25 tokens moved the probability of the correct answer from 83.33 per cent to 92.29 per cent.
Across the four filler styles, odd-length prefixes averaged a higher probability for the correct answer than even-length ones, by 3.6 to 5.8 points. Not every adjacent pair followed the pattern. Repeated measurement of eight identical inputs produced swings of up to 5.35 points. Six samples produced the wrong answer outright while still ranking the correct one higher, which is a sampling outcome rather than a reversal of preference.
The conclusion the test supports is narrow and useful: on V4.1-Flash, with a smaller cache, the positional weakness is reduced but not eliminated. It shows up as confidence that moves with input length instead of answers that invert.
Every efficient long-context model is making this trade. Compression buys context length and throughput, and it charges for them in retrieval reliability that depends on where a fact happens to sit.
Editor’s note: This is an adapted translation of the original Leiphone report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://www.leiphone.com/category/yanxishe/46BrtW83OYjXUx0b.html.