vLLM: serving LLMs on Linux with high throughput — history, installation, and first steps
If you have run a local LLM with Ollama or llama.cpp, you know it works very well — for one or two people. When demand becomes an API that must handle dozens of simultaneous requests, the situation changes: the GPU sits idle waiting, memory becomes fragmented, and throughput plummets. That is precisely the problem … Read more

