22.7-Fold Faster at Ten Million Tokens

At ten million tokens, a sparse attention layer spends 98 per cent of its prefill time deciding which tokens to look at, and almost none actually looking. The bottleneck for ultra-long context has quietly moved from attention to the index that feeds it.

Latent Sparse Attention paper figure on prefill and decode speedups
Latent Sparse Attention reports 22.7-fold and 72.4-fold prefill speedups at 10M tokens (Source: 36Kr)

Million-token context has become a default rather than a selling point. In April, DeepSeek-V4 made 1M context standard across its official services. In August, Zhipu’s GLM-5.3-Flash shipped 1M context and introduced a hybrid of sparse and linear attention to the GLM line for the first time. Both follow the same principle: each query attends to a small subset of relevant keys rather than the whole history.

That trade has been validated at 1M scale. Push to 10M and an overhead that barely registered takes over. For one global attention layer of GLM-5.3-Flash at 10M tokens, 98 per cent of prefill time went to the indexer that decides which tokens matter. Sparse attention had already shown that looking at a small subset preserves accuracy. Finding that subset still grows quadratically with context length.

A paper from London startup Prefrontal AI and ETH Zurich proposes Latent Sparse Attention, or LSA, under the title Towards Subquadratic Ultra-Long Context Attention. At 10M tokens it speeds prefill of a single GLM-5.3-Flash attention layer by 22.7-fold and of Qwen3.8-27B by 72.4-fold, with accuracy close to the original models.

Why the indexer is the problem

In full attention each query scores every preceding key, giving O(T squared) prefill and O(T) per decoded token. Sparse attention splits the job: a lightweight indexer picks the k most relevant keys, and attention runs only on those, bringing attention cost to O(Tk).

The premise holds. In an appendix test at 128k tokens, letting every Qwen3.8-27B query attend to only its genuinely top 512 keys, about 0.8 per cent of all keys, still retained 98 per cent of full-attention accuracy on RULER.

The catch is that the indexer must score every key to find those 512. DeepSeek’s DSA, MiniMax’s sparse attention, and the block-sparse schemes in GLM-5.3-Flash and Qwen all reduce the constant through lower precision, fewer dimensions or compressing four keys into one, but the indexer still grows linearly and prefill stays quadratic. The paper’s analogy: borrowing 4,096 books from a library of ten million is quick; reading every spine to choose them is not. The authors calculate that if indexer cost fell to zero, sparse-attention prefill at 10M tokens would be another 58-fold faster.

Latent Sparse Attention indexer cost breakdown at long context
At 10M tokens the indexer accounts for 98 per cent of a sparse attention layer prefill time (Source: 36Kr)

Cluster by meaning, then search a tree

Existing block-sparse methods group keys by position, packing adjacent keys into a block. That is like boxing books by arrival date with one summary per box: one important book can be drowned out by the irrelevant titles beside it.

LSA groups by content. Keys are projected into a learned low-dimensional latent space, then clustered with Ward’s hierarchical method into buckets of semantically similar keys. Each bucket keeps one centre vector and an index of the positions of its keys. A query scores only bucket centres, selects the most relevant buckets and expands them into concrete keys until k is filled. In the library analogy, books are shelved by subject rather than by acquisition date.

That Flat search gives a constant-factor gain. The Tree variant changes the complexity class: buckets are clustered again into larger node buckets, forming a hierarchy. A query starts at the top and, through beam search, keeps only the highest-scoring branches at each level, checking O(log T) centres. Indexer prefill drops from quadratic to O(T log T), which is where the subquadratic in the paper title comes from.

Two ways to deploy it

D-LSA, or Distilled-LSA, is for trained models. It freezes all original parameters, adds two small projection matrices per layer to map keys and queries into a 64-dimensional index space, then distils LSA scores to approximate the original attention distribution. The whole distillation used 352 million tokens, against 2.1 billion tokens for the dense warm-up of DeepSeek DSA’s indexer alone. A training-free variant uses principal component directions of the keys as the projection and lands close to the trained version on Qwen3.8-27B.

N-LSA, or Native-LSA, uses LSA from the start of pretraining with only the standard next-token prediction loss. It clusters the attention layer’s own keys, needs no index projection, no teacher model and no auxiliary loss, and adds no parameters.

The team wrote GPU kernels for H200 and B300 covering clustering, scoring, selection and expansion, plus a Triton version that runs on any GPU, and integrated LSA into SGLang with tensor, expert and pipeline parallelism and context parallelism across prefill and decode.

D-LSA and N-LSA architecture diagrams for flat and tree search
D-LSA and N-LSA variants of Latent Sparse Attention with flat and hierarchical bucket search (Source: 36Kr)

Accuracy and speed at 10M tokens

D-LSA was tested on Qwen3.8-27B, whose attention layers are full attention, and GLM-5.3-Flash, which ships sparse attention, pushed to 10M tokens against a 1M official limit. On MMLU-Pro, GPQA-Diamond, AIME25, MATH500 and LongBench v2, D-LSA stayed within the 95 per cent confidence interval of the baselines without materially changing reasoning length. On RULER from 4k to 10M, Qwen3.8-27B with D-LSA trailed full attention by no more than five points below 1M and then overtook it at 10M, 63.9 against 59.9. GLM-5.3-Flash with D-LSA stayed within three points under 1M, lost at most eight points between 2M and 5M and came back within four points at 10M. The comparison favours the baseline, since GLM’s native sparse attention was trained with the model while D-LSA was distilled onto frozen weights.

A new benchmark, PaperTrail, embeds fictional research threads into real 2026 arXiv papers: a fake paper proposes a result, several real papers gain paragraphs citing it, weeks later another fake paper corrects the number and renames the method, while some later papers still cite the old value. A model has to assemble the facts like a literature review, with 10M-token samples containing close to 400 to 465 papers. Qwen3.8-27B with D-LSA scored 29.1 at 10M against 19.8 for full attention.

On speed, at 10M tokens D-LSA made single-layer prefill 22.7-fold faster than GLM-5.3-Flash’s native sparse attention and 72.4-fold faster than Qwen3.8-27B’s full attention, with decode gains of 4.6-fold and 42.5-fold. The ratio widens with length: at 1M the GLM prefill gain is 3.5-fold and at 512k only 2.1-fold, because scanning a short context is cheap already. Extrapolating the trend, the Tree variant is projected at more than 90-fold over baseline sparse attention at 100M tokens.

At whole-model level the gain is diluted by unaffected layers. In SGLang deployment at 10M tokens, D-LSA cut GLM-5.3-Flash time to first token by 2.2-fold with context parallelism, from about 7.5 minutes to 3.4 minutes, and cut Qwen3.8-27B time to first token by 9.2-fold, from about 76 minutes to 8 minutes. Index memory per layer at 10M was 246 MiB for D-LSA, 331 MiB for GLM’s native scheme and 642 MiB for Qwen-style block sparsity.

N-LSA was tested at smaller scale: a roughly 500-million-parameter hybrid model, 24 layers with one attention layer in every four and Gated DeltaNet linear attention elsewhere, annealed at 128k context. Validation loss matched full attention exactly at 1.085, against 1.100 for PISA, 1.117 for NSA and 1.154 for sliding-window attention at the same sparsity budget. On RULER at 128k, N-LSA scored 32.2 against 34.7 for full attention, while NSA and PISA fell to 19.2 and 14.6. Forward plus backward of a single attention layer was 1.6-fold faster than a FlashAttention-3 full-attention implementation at 128k.

For years the long-context race was about attending to less, from fixed sliding windows to DeepSeek DSA and block-sparse schemes, and the industry has largely proved that under 1 per cent of tokens is enough. LSA points at the next constraint: once looking is cheap, finding is the expensive part. Content-organised keys were tried in Reformer and Routing Transformer, but few methods had validated both accuracy and speed above 1M tokens. As agents take on multi-day research and coding work, context windows will keep growing, and the indexer’s complexity may set the cost curve.

Editor’s note: This is an adapted translation of the original 36Kr report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://m.36kr.com/p/4073305619190657.

Leave a comment