VRAM determines the model. Dense Qwen 27B–32B, quantized Llama 70B, Mixtral 8×7B — these need 32 to 64 GB, not the 24 GB of a 4090. Engineer Donato Capitella, with Bargain Hardware, built a refurbished Supermicro with four data-center GPUs (64 GB total) for about £2.000 and benchmarked V100, P100, and MI25 in the same chassis. This article follows the video Refurbished 64GB VRAM AI Server for Local AI.
Anyone already running Ollama on a desktop hits the consumer card's ceiling. Here the bet is different: server chassis, passive card, llama.cpp compiled for the right architecture. Why the V100's Tensor Cores beat the P100 is covered in the post How graphics cards work and in the GPU vs TPU.
The chassis: 12ª generation Supermicro
The video's foundation is an enterprise Supermicro server (4U–5U tower/rack, CSE-747 family) designed for HPC, not to sit under a desk:
2× Intel Xeon Gold 6150 36 núcleos / 72 threads
64 GB DDR4 ECC Registered 4× 16 GB @ 2666 MHz
2× SSD SATA 480 GB + NVMe U.2
4× PCIe 3.0 x16 full-height, dual-slot
2× 10GbE na placa-mãe
turbinas frontais ar forçado nas aletas
fontes redundantes 1600–2000 W
Data-center Tesla and Instinct cards are passive: they have no onboard fan. The chassis blowers push air through the copper fins at thousands of RPM. A gaming desktop with 4× V100 cooks the cards — and a home PSU cannot handle the peak.
V100 vs P100 vs MI25
Three sets of four 16 GB cards (64 GB total):
| Tesla V100 16 GB | Tesla P100 16 GB | Instinct MI25 16 GB | |
|---|---|---|---|
| Architecture | Volta 12 nm (sm_70) | Pascal 16 nm (sm_60) | Vega 10 14 nm (gfx900) |
| Memory | 16 GB HBM2, 900 GB/s | 16 GB HBM2, 720–900 GB/s | 16 GB HBM2, 484 GB/s |
| Tensor Cores | yes (1ª generation, 640) | no | no |
| Precisions | FP16/32/64, INT8 | FP16/32/64 | FP16/32 |
| BF16 / FP8 | not native | not native | not native |
| Stack | CUDA / llama.cpp | CUDA / llama.cpp | ROCm / Vulkan / llama.cpp |
The dividing line is the Tensor Core. LLMs are MatMul; the V100 performs batched matrix operations. P100 and MI25 take the same path through CUDA cores / Vega SIMD — slower, cheaper. None of the three supports native BF16: the runtime converts to FP16.
A 2026 detail the video already anticipates in practice: CUDA 13 dropped sm_70. V100 is the last architecture with first-class support in CUDA 12.x. Donato's containers pin CUDA 12.6.3 and compile llama.cpp with -DCMAKE_CUDA_ARCHITECTURES=70. Toolkit 13 on the host = a binary that generates no code for the card.
Software: llama.cpp, not vLLM
Donato publishes toolboxes (Toolbx/Podman) with llama.cpp already compiled for each die:
- nvidia-v100-ai-toolboxes — CUDA 12.6.3, sm_70
- nvidia-p100-ai-toolboxes — sm_60
- mi25-gfx900-ai-toolboxes — gfx900, ROCm/Vulkan
# V100: toolbox com llama.cpp CUDA sm_70
toolbox create -c llama-v100-cuda
--image docker.io/kyuz0/nvidia-v100-ai-toolboxes:latest
toolbox enter llama-v100-cuda
llama-cli --list-devices # precisa ver as 4× 16 GB
For serving, llama.cpp wins: mature GGUF (Q4_K_M, Q4_K_XL, Q8_0), automatic layer splitting across the 4 GPUs (-ngl 99), low overhead. vLLM is the production standard on Hopper/Blackwell; on Volta/Pascal it falls back to PyTorch SDPA, AWQ barely works, and loading over PCIe 3.0 hurts.
llama-server
-m /models/qwen-27b-instruct-q4_k_xl.gguf
-ngl 99
-c 32768
-fa on
--host 0.0.0.0 --port 8080
Anyone already using the GGUF workflow without Ollama's daemon has llama-model in the same spirit: one binary, one server, no extra layer. The Docker course covers Podman/Toolbx if the host does not have it yet.
Numbers: Qwen 27B Q4
Dense model, 4-bit quantization, the setup the video uses as a coding assistant. Donato's own numbers (llama.cpp, NGL=99, flash attention):
geração (tokens/s) Qwen 27B Q4
4× V100 16 GB ~31–34 (ainda ~28 com contexto 32k)
4× P100 16 GB ~18–22
4× MI25 16 GB ~14–17
prompt processing V100
curto: ~852 tok/s
32k: ~620 tok/s
31 tok/s is interactive conversation. The P100's 20 tok/s serves background agents. MI25 loads the model at the lowest entry price — bandwidth of 484 GB/s and no matrix accelerator take their toll. Large MoE (Qwen 122B-A10B Q3 in the toolboxes README) fits in 64 GB; throughput drops, VRAM is what matters.
Everyday pain points
- Noise: 60–75 dB. Four passive cards at 250–300 W each. Not for a bedroom or under a desk. Rack, basement, garage, server room.
- PCIe 3.0. ~16 GB/s per x16 slot. Loading a 30–50 GB GGUF: 1 to 3 minutes. V100 NVLink exists, but only in specific chassis/SXM setups — the typical Bargain Hardware tower uses separate PCIe cards.
- BF16 becomes FP16. A modern model pretrained in bfloat16 loses some exponent range. Well-calibrated quantization (Unsloth Q4_K_XL etc.) reduces the pain.
- Power. 4× GPU + 2× Xeon 150 W + blowers: 800–1.200 W continuously. A suitable circuit and redundant PSU are not optional.
- CUDA 13. Do not just “update the toolkit and call it done”. Stay on the toolboxes' 12.6, or the binary ignores the V100.
Is it worth it in 2026?
If the goal is 64 GB of local VRAM for dense 27B–32B or MoE, and the budget is £2.000–3.000, 4× V100 still delivers the best return of the group: real Tensor Cores, stable llama.cpp, a ready-built machine with an industrial power supply. P100 is the middle ground. MI25 is the AMD entry point, with bandwidth limitations in mind.
It is not quiet, not PCIe 5.0, not native BF16. It is data sovereignty: the model never leaves the rack. A new 4090/5090 still wins in tok/s per watt and software (vLLM, FP8) — and still lacks 64 GB.
References
- Donato Capitella — Refurbished 64GB VRAM AI Server
- V100 toolboxes (CUDA 12.6.3 / sm_70)
- P100 toolboxes (sm_60)
- MI25 toolboxes (gfx900)
- Local AI on Linux with Ollama
- llama-model: GGUF and llama-server without the daemon
- How graphics cards work
- GPU vs TPU
Four V100s in a noisy chassis do not become an H100. They become 64 GB of VRAM that a consumer card still does not offer — and a llama.cpp that, compiled for sm_70, chats at 30 tokens per second without sending the prompt to the cloud.