The Information reported this week that at a Seed all-hands two weeks ago, Zhang Yiming spoke up in a way he rarely does and ruled out using distillation to catch the frontier large language models. His position was blunt: ByteDance should be willing to sacrifice some short-term interest for the sake of the long-term goal.
The scoreboard makes that expensive. On the latest Artificial Analysis global leaderboard, Moonshot’s Kimi K3 Max is second, Alibaba’s Qwen 3.8 Max fourth, Zhipu’s GLM 5.2 Max seventh and DeepSeek V4 Flash eighth. Chinese models hold close to half the global top ten. ByteDance’s Seed 2.1 Pro sits 21st, far below its domestic peers.
On money, ByteDance is among the most profitable internet companies in the world. On people, Seed has absorbed the best algorithm engineers from its search, advertising and recommendation systems. On compute, clusters of tens of thousands of GPUs are being built across the country. On data, Douyin and TikTok hold the largest pool of native content anywhere. Nothing is missing, and it still cannot close the gap.
Employees say the internal view is that the refusal to distil is one reason the language models trail other Chinese labs. Distillation, in plain terms, means training your model on another model’s answers. It is the accepted shortcut. There have been at least three documented arguments inside the company about whether to take it.
Three refusals
The first came in January 2025. DeepSeek-R1 landed and its reasoning style spread worldwide. Researchers inside Seed proposed following it through distillation. Zhang refused. He wanted Seed to learn how to think like a human, not to copy R1’s habit of thinking out loud.
The second came after Nvidia’s Blackwell chips were deployed. The compute gap widened, rivals had the newest silicon and ByteDance was making do with H20. The calls for distillation returned. Zhang said no again, and the company committed a further 200 billion yuan to new compute centres at Ulanqab and Huailai.
The third came when Kimi K3 broke into the global top tier. Every domestic open-source release is a stress test, and internal anxiety reached its peak. Zhang held. In his framing, leaderboards are notional and commercial sovereignty is real. People close to Seed say the ban on distilling open-source models is enforced technically, through API detection, and that mixing GPT-generated data into training sets was formally prohibited as early as April 2023.
As one executive at a rival Chinese technology group told AI Technology Review: ByteDance did distil at the start, then Zhang thought hard about it and said we should take the long road. Companies at the scale of Alibaba, Tencent and ByteDance cannot live off distilling other people.
The collapse argument
Distillation works fast. Pour in a few million high-quality instruction samples and benchmark scores can close on the teacher model within weeks. In the early phase of the race nearly everyone tried it. Since the start of 2026 Anthropic has repeatedly and publicly accused MiniMax, Moonshot, DeepSeek and Alibaba’s Tongyi lab of distilling its models, attributing the rise of Chinese models to shortcuts. Anthropic has itself been reported to have distilled OpenAI models more than once and to have trained on pirated books.
The technical case against the shortcut is model collapse. In July 2024 a joint study from Oxford, Cambridge and others made the cover of Nature, confirming for the first time that repeated training on AI-generated data erases the tail of the original data distribution and produces irreversible degradation. It behaves like inbreeding: the first generation looks fine while genetic diversity drains away, and the recessive faults arrive together a few generations later.
Meta research at ICLR 2025 pushed the finding further. One part in a thousand of synthetic data in the training mix is enough to trigger collapse, and the toxicity persists at lower ratios. Larger parameter counts amplify the effect beyond certain thresholds rather than dampening it.
Real-world knowledge is not evenly distributed. Common questions and standard answers occupy the head. Rare edge cases, counter-intuitive questions, the fine semantics of dialect and slang, and specialist knowledge in small fields live in the long tail, and that tail sets the ceiling on how a model handles the unknown. Generated answers flatten it by construction, because a model outputs the highest-probability response and filters out the odd and the uncertain. Train on that repeatedly and the model becomes more confident and narrower. Benchmark scores can still rise while its picture of the world quietly contracts.
ByteDance’s counter-asset is data. Douyin produces more than 80 million short videos a day, Toutiao recommends content at a scale of hundreds of millions, and TikTok covers more than 150 countries and regions. Cleaning petabytes of real video, text, comments and interactions is far harder than importing a few million tidy GPT-generated instructions, but only native data preserves the full distribution of the real world, including the strange and unpredictable tail.
The tokeniser nobody talks about
One detail gets overlooked: the vocabulary. A tokeniser is the model’s eyes. It decides how text is split into base units and what each unit maps to. Adopting someone else’s vocabulary means accepting their linguistic preferences and their granularity of understanding. Llama’s vocabulary is tuned for English, and Chinese idioms, internet slang and comment-stream abbreviations often shatter into four or five fragments. That raises inference cost and, more importantly, breaks the semantic links before the model has seen the sentence.
In multimodal work the problem is fatal. Seed’s core battlefield is native multimodality, not text. Seedance 2.0 already produces natively synchronised audio and video, supporting 60-second 2K clips with lip alignment in eight languages. That requires atomic-level mapping between text, video frames and audio signals in a single vector space. Distil someone else’s model and you inherit their encoding foundations: their video tokenisation, their audio-text alignment. Fine-tuning changes parameters, not the token space underneath.
Heavy industry
Training from scratch above 200 billion parameters in a mixture-of-experts configuration is a contest of engineering systems rather than single algorithms. The most punishing metric is mean time between failures. A full pre-training cycle runs for months across tens of thousands of GPUs. One memory error, one flaky high-speed link, one precision overflow in an operator can halt the run and destroy the compute spent so far.
The arithmetic is unforgiving. If a single GPU has a mean time between failures of 100,000 hours, a cluster of ten thousand cards will hit a hardware fault roughly every ten hours. Without millisecond-level fault detection, precise checkpoint resumption and complete recovery machinery, training at this scale is not attempted at all.
US export controls make it harder still. ByteDance cannot buy Nvidia’s latest Blackwell parts and is working with the restricted H20, so it has to make up the per-chip deficit with larger clusters, higher utilisation and more extreme engineering. Training clusters are now spread across Ulanqab in Inner Mongolia, Datong in Shanxi, Huailai and Zhangbei in Hebei and Wuhu in Anhui. AI infrastructure capital expenditure for 2026 has been revised up beyond 200 billion yuan, and the visible fall in net profit is driven mainly by third and fourth quarter compute purchases and data-centre construction.
Zhang himself is reported to be spending half his time on Seed. What that money and that attention buy is not one model result but a full-stack training system: operator optimisation, cluster communication scheduling, failure recovery and compute resource management, all built in house. Labs that distil never have to solve stable training across ten thousand cards, and never accumulate the capability. As parameter counts keep climbing, the barrier to entry moves from algorithms to engineering, and only the players holding the whole stack stay at the table.
Editor’s note: this English report is an adapted translation of a Chinese-language original published by LeiPhone (leiphone.com). Figures, dates and direct quotations follow the source.