DeepSeek: The Chinese AI Company Upending the Industry with Open-Source LLMs

Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co., Ltd., commonly known as DeepSeek, is a rising force in the artificial intelligence landscape. Founded in July 2023 and based in Hangzhou, Zhejiang, China, DeepSeek focuses on developing open-source large language models (LLMs). Backed by the Chinese hedge fund High-Flyer, DeepSeek has quickly gained recognition for its cost-effective AI solutions that rival those of industry giants like OpenAI and Meta.

A Brief History: From Hedge Fund to AI Pioneer

Founding and Early Years (2016–2023)

The story of DeepSeek begins with High-Flyer, co-founded in February 2016 by Liang Wenfeng, an AI enthusiast with a background in trading. High-Flyer initially focused on stock trading, leveraging GPU-dependent deep learning models. By 2019, the company became a hedge fund dedicated to AI trading algorithms, heavily relying on Nvidia chips.

Recognizing the potential of AI beyond finance, Liang established an artificial general intelligence (AGI) lab in April 2023, separate from High-Flyer's core business. This lab officially became DeepSeek in July 2023, with High-Flyer as its primary investor. Despite reluctance from venture capital firms, DeepSeek forged ahead, driven by a vision to create accessible and powerful AI models.

Model Releases (2023–Present)

DeepSeek rapidly introduced a series of models:

• November 2023: DeepSeek Coder, designed for code generation and understanding.
• November 2023: DeepSeek-LLM series, general-purpose language models.
• January 2024: DeepSeek-MoE models, incorporating a mixture of experts for enhanced performance.
• April 2024: DeepSeek-Math models, specialized for mathematical reasoning.
• May 2024: DeepSeek-V2, featuring architectural improvements and extended context length.
• June 2024: DeepSeek-Coder V2 series, refining code-generation capabilities.
• December 2024: DeepSeek-V3, building upon V2 with multi-token prediction.
• January 2025: DeepSeek-R1, a reasoning-focused model with comparable performance to GPT-4o.

This rapid development cycle highlights DeepSeek's commitment to innovation and its ability to quickly adapt to the evolving AI landscape.

Company Operations and Strategy

DeepSeek operates from Hangzhou, Zhejiang, with funding and ownership primarily held by High-Flyer. Liang Wenfeng serves as the CEO of both companies, maintaining a significant stake in DeepSeek through shell corporations.

The company's strategy centers on research and development. Unlike some competitors, DeepSeek has not publicly emphasized near-term paths to commercialization. This may allow it to avoid stringent AI regulations within China. DeepSeek's hiring practices prioritize technical abilities and diverse knowledge backgrounds, seeking talent from top Chinese universities and fields outside computer science.

Training Framework: The Power Behind DeepSeek's Models

DeepSeek utilizes powerful computing clusters, including Fire-Flyer and Fire-Flyer 2, to train its models. Fire-Flyer 2 features a co-designed software and hardware architecture:

• Nvidia GPUs with high-speed interconnects (200 Gbps)
• Two-zone cluster design for cross-zone task support
• Fat tree network topology for high bisection bandwidth

The software side includes:

• 3FS (Fire-Flyer File System): A distributed parallel file system optimized for asynchronous random reads using Direct I/O and RDMA Read.
• hfreduce: A library for asynchronous communication, designed as an alternative to Nvidia's NCCL, primarily used for allreduce operations during backpropagation.
• hfai.nn: A software library of commonly used operators for neural network training, similar to PyTorch's torch.nn.
• HaiScale Distributed Data Parallel (DDP): A parallel training library supporting various parallelism forms such as data parallelism, pipeline parallelism, and tensor parallelism.
• HAI Platform: Applications for task scheduling, fault handling, and disaster recovery.

These technologies enable efficient training and optimization of DeepSeek's large language models.

Exploring DeepSeek's Models

DeepSeek offers a diverse range of LLMs, each with unique characteristics and capabilities. The company's models are generally "open weight," which allows more access than closed proprietary systems, but less freedom than truly open source systems.. Here's an overview:

DeepSeek Coder

• Designed for code generation and understanding, with 8 models (4 Base, 4 Instruct).
• 16K context lengths.
• Trained on 1.8T tokens (87% source code, 10% code-related English, 3% code-unrelated Chinese).
• Open-source code under MIT License.

DeepSeek-LLM

• General-purpose language models with 7B and 67B parameters.
• Benchmark results exceeding many open-source LLMs.
• Trained on 2 trillion tokens of English and Chinese text.
• Available in Base and Chat versions.
• Architecturally similar to the Llama series.

DeepSeek-MoE

• Mixture of Experts (MoE) models with 16B parameters (2.7B activated per token).
• Performance comparable to a 16B MoE as a 7B non-MoE.
• Employs "shared experts" for core knowledge and "routed experts" for specialized information.

DeepSeek-Math

• Models specialized for mathematical reasoning (Base, Instruct, RL).
• Initialized with DeepSeek-Coder Base v1.5 7B.
• Trained using Supervised Finetuning (SFT) and Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO).

DeepSeek V2

• Introduced in May 2024, featuring architectural innovations like multi-head latent attention (MLA).
• Includes DeepSeek-V2 and DeepSeek-V2 Lite (Base) and Chat versions.
• Extended context length to 128K using YaRN.
• Trained on 8.1T tokens with a higher proportion of Chinese tokens.

DeepSeek V3

• Builds upon V2 with multi-token prediction for faster decoding.
• Trained on 14.8T tokens of multilingual data.
• Achieved performance comparable to GPT-4o and Claude 3.5 Sonnet in benchmarks.

DeepSeek R1

• DeepSeek-R1-Lite-Preview was trained for logical inference, mathematical reasoning, and real-time problem-solving.
• DeepSeek-R1 and DeepSeek-R1-Zero were initialized from DeepSeek-V3-Base.
• DeepSeek-R1-Zero was trained exclusively using GRPO RL with rule-based rewards.

Significance and Impact

DeepSeek's emergence has significantly impacted the AI industry, challenging established players and driving down costs. Its ability to achieve comparable performance to larger models with significantly lower training costs has been described as "upending AI."

Domestically, DeepSeek is dubbed the "Pinduoduo of AI," sparking a price war among Chinese tech giants. Despite its low prices, DeepSeek has maintained profitability, unlike some of its competitors.

Overseas, DeepSeek's development amid US sanctions highlights its resilience and ability to innovate despite restrictions on access to advanced chips.

Controversies and Challenges

DeepSeek faces scrutiny regarding content moderation, with reports indicating adherence to local regulations limiting responses on sensitive topics. Uncensored models have shown bias towards Chinese government viewpoints.

It has been banned on goverment devices in South Korea,Australia and Taiwan.

Conclusion

DeepSeek's rapid rise as a developer of open-source large language models marks an important new stage in the evolution and accessibility of AI. Fueled by an ethos of innovation and technical excellence, DeepSeek is well-positioned to continue pushing the boundaries of what's possible in artificial intelligence.