Large Language Models (LLMs) are revolutionizing how we interact with information. Imagine having a powerful LLM, customized with your own knowledge base, running entirely on your local machine. This article guides you through setting up DeepSeek R1 with Ollama and AnythingLLM, allowing you to create a personalized and private AI assistant.
DeepSeek R1 stands out as a powerful open-source LLM. It offers different model sizes, catering to various hardware capabilities. When exploring LLMs, it's helpful to understand the concept of "distillation." The original article mentions "671B" as the "base model”. Distillation involves taking a large, complex model (like a 671B parameter model) and creating smaller, more efficient versions (such as 1.5B, 7B, or 8B parameter models). These distilled models, like those derived from Qwen (通义千问) or Llama, retain much of the original model's knowledge while requiring less computational power.
Here's a breakdown of the DeepSeek R1 model sizes and their requirements, as mentioned in the original article:
| Model Size | Minimum GPU Memory | Recommended GPU | CPU Memory | Use Case |
|---|---|---|---|---|
| 1.5B | 4GB | RTX 3050 | 8GB | Personal Learning |
| 7B/8B | 16GB | RTX 4090 | 32GB | Small Projects |
| 14B | 24GB | A5000 x2 | 64GB | Professional Use |
| 32B | 48GB | A100 40GB x2 | 128GB | Enterprise Service |
| 70B | 80GB | A100 80GB x4 | 256GB | High-Performance Computing |
| 671B | 640GB+ | H100 Cluster | N/A | Supercomputing/Cloud Computing |
| Note: This information is based on recommendations and may vary depending on your specific setup and usage. |
This article will focus on setting up a smaller model size suitable for personal use and those with limited GPU resources.
Before we begin, ensure your system meets the following requirements:
Ollama is a powerful tool that simplifies running LLMs locally. It handles the complexities of managing dependencies and configurations, making it easy to get started.
ollama command from any terminal window.
C:\Program Files\Ollama).ollama. If Ollama is installed correctly, you should see a list of available commands.Now that Ollama is installed, you can download the DeepSeek-R1 model.
Pull the Model: In your terminal, run the following command to download the 8B version of the DeepSeek-R1 model:
ollama run deepseek-r1:8b
This command will download the model weights. The first time you run this, it might take a while, depending on your internet speed.
List Installed Models: To confirm the model has been downloaded successfully, run the following command:
ollama list
This will display a list of all models currently installed by Ollama.
Run the Model: Once downloaded, you can run the model directly from the command line:
ollama run deepseek-r1:8b
This will start the DeepSeek-R1 model in interactive mode, allowing you to start asking questions and testing its capabilities.
AnythingLLM provides a user-friendly interface for interacting with your local LLM and building knowledge bases.
AnythingLLM allows you to upload documents and create a knowledge base that the LLM can use to answer your questions.
To effectively use DeepSeek R1 with your knowledge base, you need to connect it within AnythingLLM and configure appropriate chat prompts.
Select Ollama as your Model Provider: In AnythingLLM's settings, choose Ollama as the model provider. This tells AnythingLLM to use the local Ollama installation to run the LLM.
Specify the Model: Select "deepseek-r1:8b" (or the version you downloaded) as the model to use.
Customize Chat Prompts: This is a crucial step. You need to create effective chat prompts that instruct the LLM on how to use the knowledge base. For example, you could use a prompt like:
"You are a helpful research assistant. Use the provided documents to answer the following question. If the answer is not found in the documents, respond that you cannot answer based on the provided information."
Experiment with different prompts to see what works best for your specific use case. Clear and specific prompts will lead to more accurate and relevant answers.
With everything set up, you can now start asking questions and leveraging your local LLM knowledge base. In the AnythingLLM interface, type your question and submit it. The LLM will use the uploaded documents and the chat prompt to generate an answer.
By following these steps, you can create a powerful and private LLM knowledge base on your local machine, enabling you to explore, learn, and innovate with AI.
