DeepSeek: A Novel LLM-Powered Retrieval Engine for Comprehensive Entity Collection

In the rapidly evolving landscape of AI-driven information retrieval, a new approach is emerging that moves beyond simply answering questions. This approach focuses on comprehensively collecting and organizing information, and DeepSeek is at the forefront. Developed by dzhng and open-sourced on GitHub, DeepSeek is an LLM-powered retrieval engine designed to meticulously process vast amounts of data to compile exhaustive lists of entities.

Unlike traditional "answer engines" that aim to provide a single, correct answer by aggregating sources, DeepSeek functions as a true retrieval engine. It sifts through numerous sources, identifies relevant entities, and then enriches them with associated data. This results in a structured table of information, offering users a comprehensive overview of the topic at hand.

Answer Engine vs. Retrieval Engine: A Key Distinction

To understand the significance of DeepSeek, it's crucial to differentiate between answer engines and retrieval engines:

• Answer Engine: Focuses on aggregating information from various sources to find the most accurate answer to a specific question. Examples include platforms like Perplexity AI, GPT-Researcher, and Aomni. The output is typically a research report or concise answer.
• Retrieval Engine: Employs LLMs to meticulously process numerous sources and create comprehensive lists of all relevant entities. DeepSeek is a prime example, delivering a structured table with enriched columns for each entity.

How DeepSeek Works: A Multi-Step Research Pipeline

DeepSeek's architecture is based on a multi-step research agent, also known as "flow engineering." The process involves breaking down the initial user query into a plan and iteratively constructing the answer as it flows through the system. The research pipeline consists of four key steps:

1. Plan: Based on the user's query, the planner defines the scope of the end result, identifying the type of entity to extract and defining relevant columns for the resulting table. These columns represent additional data points related to the entities.

2. Search: DeepSeek utilizes both standard keyword search and neural search (powered by Exa) to locate relevant content. Keyword search excels at finding user-generated content like reviews and listicles, while neural search is adept at identifying specific entities like companies or research papers.

3. Extract: LLMs are used to process the content found during the search phase and extract specific entities and their associated information. A novel technique involving special tokens to define the range of content to extract ensures speed and efficiency.

4. Enrich: A smaller "answer agent" within DeepSeek enriches all the columns defined by the planner for each entity. This step is the most time-consuming but ensures that the resulting table is thorough and informative.

Getting Started with DeepSeek

To use DeepSeek, you'll need to install it using a package manager like npm, yarn, pnpm, or bun. Detailed installation instructions can be found in the Install documentation.

After installation, you can run the development server and explore pre-built examples. To fully utilize DeepSeek, you'll need an API key for both Anthropic and Exa. These keys should be stored in a .env file like this:

ANTHROPIC_KEY="your_anthropic_api_key"
EXA_KEY="your_exa_api_key"

Future Enhancements for DeepSeek

The developers of DeepSeek are actively working on improvements and new features, including:

• Sorting and Ranking: Implementing a sorting and ranking mechanism to prioritize retrieved entities by relevance, especially for queries with qualifiers like "best" or "newest."
• Improved Entity Resolution: Enhancing the agent's ability to detect and resolve duplicate entities, which can occur when entities have slightly different names or representations. Referencing resources like this article on product categorization for better entity title formatting.
• Enhanced Source Verification: Adding better verification processes to ensure that information extracted from sources is genuinely connected to the original entity.
• Deep Source Browsing: Enabling the agent to navigate within web pages to extract relevant information from deeper levels of content, crucial for tasks like searching research papers on platforms like arXiv.
• Real-time Data Streaming: Implementing a streaming mechanism to populate the list of entities and enrich cells in real-time within the user interface, providing a more dynamic and informative user experience.

Contribute to DeepSeek

DeepSeek represents a significant step forward in AI-powered information retrieval. By providing a comprehensive, structured overview of entities, it empowers users to gain deeper insights and make more informed decisions.

If you're interested in contributing to DeepSeek or discussing ideas, you can reach out to the developer via email or Twitter. The project is open-source and welcomes contributions from the community. By collaborating and sharing use cases, we can collectively push the boundaries of LLM-powered retrieval engines and unlock new possibilities for knowledge discovery.