Kimi K3, unpacked: 896 experts and the trade-offs behind them
Su Jianlin, a Kimi researcher and the author of rotary position embeddings, has published a short paper titled Notes on the MoE and Attention of K3. It concentrates on the trade-offs Moonshot made in expert architecture, training stability, load balancing and attention design. K3 carries 2.8 trillion total parameters and 896 routed experts. The model … Read more