Installing Kimi K3 Locally: A Comprehensive Step-by-Step Guide

System Requirements

Kimi K3 uses a Mixture-of-Experts (MoE) architecture with billions of active parameters. Running it locally requires high-bandwidth VRAM and unified memory allocation.

Recommended System Specifications

• GPU: Dual NVIDIA RTX 4090 (48GB total VRAM) or single RTX A6000.
• RAM: Minimum 64GB DDR5 system memory.
• Storage: 150GB free space on a NVMe M.2 SSD.
• OS: Linux (Ubuntu 22.04 LTS or newer recommended) or Windows Subsystem for Linux (WSL2).

Step 1: Set Up Your CUDA Environment

Ensure you have CUDA Toolkit 12.1 or newer installed on your host system.

Verify your installation:

nvcc --version

If it is missing, download the appropriate binaries from the official NVIDIA developer portal.

Step 2: Install vLLM Container or Library

The recommended inference engine for Kimi K3 is vllm because of its PagedAttention and tensor parallel processing capabilities.

Create a virtual environment:

python3 -m venv kimi-env
source kimi-env/bin/activate
pip install --upgrade pip
pip install vllm

Step 3: Run the Model

Run the server using a multi-GPU configuration to split the parameter weights:

python3 -m vmm.entrypoints.openai.api_server \
  --model Kimi-K3-MoE \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --port 8000