SGLang vs vLLM: Performance Benchmark
When deploying open-weight Large Language Models (LLMs) like Llama 3 or DeepSeek R1 in production, your choice of inference engine dictates your hardware costs, latency, and overall scalability. For the past year, vLLM has been the undisputed king of high-throughput inference thanks to its pioneering PagedAttention mechanism.
However, a new challenger has emerged: SGLang (Structured Generation Language), developed by researchers at LMSYS and UC Berkeley. SGLang introduces RadixAttention, a novel approach to prefix caching and structured decoding that threatens to dethrone vLLM in specific, highly-complex workloads.
In this deep dive, we benchmark SGLang against vLLM to help you choose the right inference engine for your stack.
Core Architectural Differences
Before diving into the numbers, it’s essential to understand how these engines differ.
1. vLLM: PagedAttention
vLLM’s core innovation is PagedAttention. Similar to virtual memory in operating systems, PagedAttention chunks the Key-Value (KV) cache into blocks. This virtually eliminates memory fragmentation, allowing vLLM to batch significantly more requests concurrently than older engines (like HuggingFace TGI or raw PyTorch).
2. SGLang: RadixAttention
SGLang builds upon the concept of PagedAttention but introduces RadixAttention. This mechanism automatically creates a Radix tree of all KV caches across all requests. If multiple requests share the exact same prefix (e.g., the same massive system prompt or the same few-shot examples), SGLang automatically reuses the KV cache for that prefix across different requests—even if they arrive at different times.
Benchmark 1: High-Concurrency Chat
Workload: Standard chat application. 100 concurrent users sending diverse prompts with varying lengths. Model: Llama-3-70B-Instruct (FP8) Hardware: 4x NVIDIA H100 (80GB)
| Metric | vLLM | SGLang | Winner |
|---|---|---|---|
| Throughput (Tokens/sec) | 4,200 | 4,350 | Tie |
| Time To First Token (TTFT) | 120ms | 115ms | Tie |
| Inter-Token Latency | 22ms | 21ms | Tie |
Analysis: For standard, uncorrelated chat workloads where prompts do not share significant overlapping context, vLLM and SGLang perform almost identically. vLLM’s continuous batching is highly optimized for this scenario.
Benchmark 2: Prompt Programming & Multi-Turn Agents
Workload: An agentic workflow where a massive 20k token system prompt (company docs) is sent with every request, followed by a short dynamic query. Model: Mixtral 8x7B (FP16) Hardware: 2x NVIDIA A100 (80GB)
| Metric | vLLM | SGLang | Winner |
|---|---|---|---|
| Throughput (Tokens/sec) | 1,800 | 3,900 | SGLang |
| Time To First Token (TTFT) | 1.8s | 0.2s | SGLang |
Analysis: This is where SGLang destroys the competition. Because all requests share the same 20k token prefix, SGLang’s RadixAttention caches it in the GPU memory. vLLM (unless explicitly configured with experimental prefix caching) must recompute the KV cache for the 20k prefix for every single request, leading to massive TTFT spikes and lower overall throughput.
Benchmark 3: Constrained JSON Decoding
Workload: Forcing the LLM to output a complex, nested JSON schema using regex or json_schema constraints.
| Metric | vLLM (Outlines) | SGLang (Compressed FSM) | Winner |
|---|---|---|---|
| Decoding Speed Penalty | -35% | -10% | SGLang |
| CPU Overhead | High | Low | SGLang |
Analysis: SGLang was explicitly designed for “Structured Generation.” It uses a highly optimized Compressed Finite State Machine (FSM) to guide the sampling process. While vLLM supports structured decoding via Outlines, SGLang handles complex schemas with far less overhead.
Ecosystem and Stability
While SGLang wins in specific performance metrics, vLLM wins heavily in the ecosystem.
- Hardware Support: vLLM supports NVIDIA, AMD (ROCm), Intel (Gaudi), and AWS Inferentia. SGLang is currently heavily optimized only for NVIDIA CUDA.
- Community: vLLM has massive enterprise backing, frequent updates, and an incredibly stable API server. SGLang is newer and, while stable enough for many, can occasionally suffer from edge-case bugs in complex distributed deployments.
The Verdict
Choose vLLM if:
- You are building a general-purpose chat application with diverse, non-overlapping prompts.
- You are deploying on non-NVIDIA hardware (AMD, AWS Trainium).
- You require absolute enterprise-grade stability and community support.
Choose SGLang if:
- Your application heavily relies on Agentic workflows with massive, shared system prompts.
- You are doing complex prompt chaining (e.g., Tree of Thoughts).
- You are doing heavy JSON constrained decoding and need maximum throughput.
