DeepSeek-R1 Architecture: How Mixture of Experts (MoE) & Reasoning Fine-Tuning Scale Intelligence

The Architecture That Shook Open Source AI

DeepSeek-R1 represents a monumental leap forward in open-weight reasoning models. By combining Multi-head Latent Attention (MLA) with a 210B parameter Mixture-of-Experts (MoE) backbone where only 21B parameters activate per token, DeepSeek-R1 achieves reasoning performance comparable to OpenAI o1 at a fraction of training and inference costs.

1. Multi-head Latent Attention (MLA)

Traditional Multi-Head Attention (MHA) allocates separate Key and Value vectors per head, consuming massive GPU VRAM during KV-cache generation. MLA compresses Keys and Values into a shared low-rank latent vector.

References & Further Reading:

• DeepSeek-R1 Technical Report (arXiv:2501.12948)
• vLLM Engine Specs for Sparse MoE Serving