vLLM CUDA Out of Memory (OOM): Fixes for max_model_len, gpu_memory_utilization, and PagedAttention

Serving large language models like DeepSeek R1 70B or Llama 3 with vLLM provides high token throughput. However, engineers frequently encounter torch.cuda.OutOfMemoryError failures during engine startup or peak concurrent user traffic.

This tutorial provides tested configuration flags and memory allocation strategies to eliminate vLLM CUDA OOM crashes permanently.

Memory Allocation Architecture

 ┌────────────────────────────────────────────────────────────────────────┐
 │                    VLLM GPU VRAM ALLOCATION MATRIX                     │
 └───────────────────────────────────┬────────────────────────────────────┘
                                     │
 ┌──────────────────────────┐        │        ┌──────────────────────────┐
 │ MODEL WEIGHTS (FP16/FP8) ├────────┼───────►│ PAGEDATTENTION KV CACHE  │
 │ • Static Model Memory    │        │        │ • Dynamic Request Blocks │
 └──────────────────────────┘        │        └──────────────────────────┘
 ┌──────────────────────────┐        │
 │ PYTORCH & CUDA OVERHEAD  ├────────┘
 │ • Reserved Buffer (10%)  │
 └──────────────────────────┘

3 Core Configuration Fixes

Fix 1: Adjust --gpu-memory-utilization

If vLLM crashes during engine initialization before serving any requests, reduce the reserved KV cache threshold from 0.90 to 0.82:

python3 -m vllm.entrypoints.openai.api_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
    --gpu-memory-utilization 0.82 \
    --tensor-parallel-size 4

Fix 2: Cap Context Window (--max-model-len)

Large models defaulting to 128k context windows require massive KV cache memory. Restrict maximum sequence length to match actual application requirements:

--max-model-len 8192

Fix 3: Enable FP8 / INT4 KV Cache Quantization

Reduce KV cache VRAM footprint by 50% without retraining:

--kv-cache-dtype fp8