vLLM: serving LLMs on Linux with high throughput — history, installation, and first steps

If you've already run a local LLM with Ollama or llama.cpp, you know it works very well — for one or two people. When the demand becomes an API that needs to serve dozens of simultaneous requests, the game changes: the GPU sits idle waiting, memory gets fragmented, and throughput plummets. That's exactly the problem that vLLM solves, and that's why it has become the standard inference engine for much of the industry.

This post covers the project's history, the technical idea that made it fast, and a step-by-step walkthrough to install and launch your first server on Linux.

Where vLLM came from

vLLM was born at the Sky Computing Lab at UC Berkeley. The repository was created in February 2023, and the public release came in 20 June 2023, in a post announcing something hard to believe at the time: up to 24x more throughput than HuggingFace Transformers and up to 3,5x more than Text Generation Inference, without changing the model's architecture.

It wasn't lab marketing. Before the announcement, vLLM had already been running in production for two months serving the Vicuna and the Chatbot Arena from LMSYS — it was the technology that allowed a small research group to pay the GPU bill for a public chatbot.

The core idea was published in the paper “Efficient Memory Management for Large Language Model Serving with PagedAttention”, presented at SOSP 2023 — which says a lot about the nature of the work: it's not a machine learning paper, it's a paper about operating systems.

The timeline since then:

• Feb/2023 — repository created on GitHub, under Apache license 2.0.
• Jun/2023 — public release; already in production on Vicuna and Chatbot Arena.
• Oct/2023 — PagedAttention paper at SOSP.
• Jan/2025 — vLLM V1, core rewrite (scheduler, KV cache manager, worker, sampler, and API server), with ~1,7x gain. Released in alpha in v0.7.0 and becomes the default engine in v0.8.0.
• May/2025 — the project is now hosted by PyTorch Foundation, one of the first platform projects under this umbrella.
• Aug/2026 — version v0.28.0. The repository has over 91 thousand stars and 21 thousand forks, with hundreds of contributors.

PagedAttention in 30 seconds

To generate each new token, the model needs to query the attention keys and values of all previous tokens. This material resides in VRAM and is called KV cache. It is large (reaching 1,7 GB per sequence in LLaMA-13B) and, worse, dynamic: the size depends on the conversation length, which no one knows in advance.

Previous engines solved this by reserving a contiguous block of the maximum possible size for each request. Result: the authors measured 60% to 80% of waste of VRAM from fragmentation and over-provisioning. Wasted memory means fewer concurrent requests, which means idle GPU.

PagedAttention applies to the KV cache the same solution the kernel has used for system memory for decades — paging:

• the KV cache of each sequence is divided into blocks of fixed size;
• the blocks don't need to be contiguous in VRAM, and are allocated on demand;
• a block table maps logical blocks to physical ones, exactly like a page table.

The analogy is literal: blocks are pages, tokens are bytes, sequences are processes. And, as in any system with paging, you get for free the sharing: two requests with the same prompt point to the same physical blocks, with reference counting and copy-on-write. Waste drops to less than 4%, and there's leftover VRAM to group many more sequences in the same batch.

Add to that the continuous batching — instead of waiting for the entire batch to finish, the scheduler slots in new requests as soon as a spot opens — and you have the full explanation of the throughput gain.

vLLM, Ollama, or llama.cpp?

All three serve LLMs, but they solve different problems. Choosing wrong hurts:

	vLLM	Ollama / llama.cpp
Use case	Multi-user API, production, high throughput	Personal use and small teams; they handle concurrency (--parallel, OLLAMA_NUM_PARALLEL), but throughput drops quickly under load
Hardware	Dedicated GPU with spare VRAM	Runs on CPU, partial GPU, little VRAM
Model format	Hugging Face weights (safetensors)	Quantized GGUF
Model doesn't fit in VRAM	Offload to RAM is possible (--cpu-offload-gb), but it's costly in latency	Offloads to RAM and continues
Ease of use	Requires planning VRAM and parameters	One command and it works

Practical rule: if it's just you alone on your desktop, use Ollama or the llama-server. If it's a service that multiple people or multiple agents will consume at the same time, vLLM.

Requirements

• Linux (it is the first-class platform) and Python 3.10 to 3.13.
• NVIDIA GPU with driver and CUDA working is the smoothest path. There's support for AMD (ROCm), Intel (XPU), Google TPUs and Ascend NPUs, in addition to the vLLM-Metal for Apple Silicon.
• VRAM: account for the model weights plus the KV cache. A 7B model in FP16 requires ~14 GB just for weights — add headroom for the cache. Things get comfortable starting at 24 GB.

Check the environment first:

nvidia-smi
python3 --version

Installing

The recommended way by the documentation is with uv, which resolves the correct PyTorch index for your CUDA version automatically:

# instala o uv, se ainda não tiver
curl -LsSf https://astral.sh/uv/install.sh | sh

# ambiente isolado + vLLM
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The --torch-backend=auto inspects the installed CUDA driver and picks the right build. To pin a specific version, replace with --torch-backend=cu129 (or the build you use — vLLM binaries today come out for CUDA 12.8, 12.9, and 13.0).

Just want to try without creating any environment?

uv run --with vllm vllm --help

On AMD GPUs, the index is different:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

The --python 3.12 it's not a detail: ROCm wheels only exist for that version. With 3.11 or 3.13 the installer silently falls back to the CUDA wheel from PyPI, and the error only appears at runtime, like libcudart.so: cannot open shared object file.

And if you prefer not to install anything on the host, the official image handles it:

docker run --runtime nvidia --gpus all 
  -v ~/.cache/huggingface:/root/.cache/huggingface 
  -p 8000:8000 --ipc=host 
  vllm/vllm-openai:latest 
  --model Qwen/Qwen2.5-1.5B-Instruct

The --ipc=host gives the container access to the host's shared memory. Without it, /dev/shm it stays lowercase and vLLM breaks when initializing the workers — especially with tensor parallel. The alternative, if you don't want to share the IPC namespace, is --shm-size=8g.

First test: batch inference

Before starting the server, it's worth doing an offline test to confirm the GPU is being used. Create teste.py:

from vllm import LLM, SamplingParams

prompts = [
    "O kernel Linux foi criado por",
    "A capital do Brasil é",
    "Em uma frase, explique o que é paginação de memória:",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=64)

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f">> {output.prompt!r}")
    print(f"   {output.outputs[0].text!r}n")

python3 teste.py

Notice that the three prompts are processed together, not in sequence — that's batching in action. A detail that catches a lot of people: llm.generate() doesn't apply the chat template of the model. For models instruct, use llm.chat() with the same message structure as the OpenAI API:

mensagens = [[{"role": "user", "content": p}] for p in prompts]
outputs = llm.chat(mensagens, sampling_params)

Launching the server compatible with the OpenAI API

This is the main reason for vLLM's existence in production: it speaks the OpenAI protocol, so any application, SDK or agent that already uses OpenAI can point to it by just changing the base_url.

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server starts on http://localhost:8000. Test:

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Explique inodes em duas frases."}],
    "temperature": 0.7
  }'

On the Python client side, it's the official OpenAI SDK without hacks:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

resposta = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Escreva um one-liner que conta arquivos por extensão."}],
)
print(resposta.choices[0].message.content)

To require authentication, start with --api-key SEGREDO — or, better, with the variable VLLM_API_KEY, because a key on the command line appears in the ps aux. The server accepts multiple keys, which facilitates rotation without downtime.

Attention: o --api-key authenticates only the prefixes /v1, /v2 and /inference. Endpoints like /invocations (which exposes the same inference capability), /pooling, /classify and the control /pause, /resume and /abort_requests remain open. If you use --host 0.0.0.0, put vLLM behind a reverse proxy that only exposes the routes you want to expose — never leave port 8000 directly accessible from the network.

The parameters that really matter

The default works, but it's almost never what you want in production:

vllm serve Qwen/Qwen2.5-7B-Instruct 
  --host 0.0.0.0 --port 8000 
  --gpu-memory-utilization 0.90 
  --max-model-len 8192 
  --tensor-parallel-size 1 
  --api-key MINHA_CHAVE

• --gpu-memory-utilization — fraction of VRAM that vLLM can occupy (default 0.92 in current versions; it was 0.9 until v0.11). Lower it if the GPU also runs your monitor or another process; raise it if the card is dedicated. The higher, the more KV cache, the more simultaneous requests.
• --max-model-len — maximum context window. It's parameter #1 to resolve memory error at startup: models advertise huge contexts (128k) that require KV cache you don't have. Cut to what your application actually uses.
• --tensor-parallel-size — number of GPUs to split the model across. Use when the model doesn't fit on a single card.
• --dtype — auto works well; bfloat16 on Ampere or newer cards, float16 on older cards.
• --quantization — for AWQ, GPTQ, or FP8 weights, which cut VRAM usage in half or more.
• --served-model-name — the name that the API exposes, useful for not leaking the Hugging Face path to clients.
• --cpu-offload-gb — how many GiB of weights to push to RAM, per GPU. Saves the day when the model doesn't fit, but each forward pass goes through the bus: only use if the alternative is not running.

Models with restricted licenses (Llama, Gemma) require a Hugging Face token:

export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx
vllm serve meta-llama/Llama-3.1-8B-Instruct

Running as a service in systemd

On a real server, vLLM has to come up on its own at boot. Create /etc/systemd/system/vllm.service:

[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm
Environment="HF_HOME=/opt/vllm/hf"
Environment="VLLM_API_KEY=MINHA_CHAVE"
ExecStart=/opt/vllm/.venv/bin/vllm serve Qwen/Qwen2.5-7B-Instruct 
  --host 0.0.0.0 --port 8000 
  --gpu-memory-utilization 0.90 
  --max-model-len 8192
Restart=always
RestartSec=10
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

The TimeoutStartSec high is intentional: on first run the model is downloaded from Hugging Face and the kernels are compiled, and with the default of 90 seconds systemd would give up midway. Notice it only has effect because the unit uses Type=exec — with Type=simple, systemd considers the service started right at fork() and the startup timeout never gets a chance to matter.

The service runs with its own user and environment — the .venv that you created in your home directory doesn't work here:

sudo useradd -r -s /usr/sbin/nologin -d /opt/vllm vllm
sudo mkdir -p /opt/vllm/hf
sudo chown -R vllm:vllm /opt/vllm

# instala o vLLM dentro de /opt/vllm, como o usuário do serviço
sudo -u vllm uv venv --python 3.12 --seed /opt/vllm/.venv
sudo -u vllm env VIRTUAL_ENV=/opt/vllm/.venv uv pip install vllm --torch-backend=auto

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
journalctl -u vllm -f

Common problems

• CUDA out of memory at startup — it's almost always the KV cache, not the weights. Reduce --max-model-len first, then --gpu-memory-utilization, and only then switch models or move to quantized weights.
• Error in /dev/shm or workers dying in Docker — it's missing --ipc=host.
• Model responds nonsense in chat mode — you used generate() instead of chat(), and the chat template wasn't applied.
• Service deployment takes too long — is the compilation of kernels and capture of CUDA graphs, which happen during engine initialization, not on the first request. The artifacts are stored in ~/.cache/vllm and are reused on subsequent boots; in Docker, mount a volume at this path to avoid recompiling for each container. To skip this step, at the cost of slower decode, use --enforce-eager.
• Download stuck or 401 — restricted license model: accept the terms on the Hugging Face page and export HF_TOKEN.

Closing

vLLM is the rare example of a research project that became standard infrastructure because it solved the right problem: it treated a GPU's VRAM the way a kernel treats a server's RAM. The lesson is old — pagination, page tables, copy-on-write — only the hardware is new.

To continue: understand the difference between GPU and TPU, see how to set up a refurbished AI server with 64 GB of VRAM to host all this without breaking the bank, or compare with the simplest path to run local AI with Ollama.

Official links: vllm.ai · github.com/vllm-project/vllm · documentation · PagedAttention paper