llama.cpp: getting to know, installing and running local AI on Linux

There's a project that almost everyone depends on without knowing. When you run a local model with Ollama, with LM Studio, with Jan or with that AI app on your phone, it's very likely that, underneath, the one doing the math is the llama.cpp — an inference engine written in pure C++, without Python, without mandatory CUDA, without any dependencies.

This post is the straightforward guide: where it came from, how to install, how to run the first model and how to make it serve an API.

Where llama.cpp came from

The story begins with a library, not with LLaMA. In September 2022, the Bulgarian Georgi Gerganov started writing GGML — a tensor algebra library in C, made to run fast on common CPU.

In March 2023 the LLaMA weights from Meta leaked. A few days later, on 10 March 2023, Gerganov published llama.cpp: Meta's model running on a MacBook, no GPU, no Python, nothing. It was one of those moments where the ruler of what is possible shifts — until then, “running an LLM” meant renting datacenter GPU.

What came next:

• June 2023 — Gerganov founds the ggml.ai in Sofia, Bulgaria, with pre-seed investment from Nat Friedman (former CEO of GitHub) and Daniel Gross, via AI Grant.
• 21 of August 2023 — the GGUF, format is born, which replaces the old GGML and becomes the de facto standard for quantized models.
• February 20 of 2026 — a Hugging Face announces that Gerganov and the GGML team have joined the organization as full-time employees, maintaining autonomy and technical leadership over the project.
• July 2026 — major restructuring: the llama-cli stops running inference in its own process and becomes a thin client that launches the llama-server underneath and communicates with it.

Today the project lives at ggml-org/llama.cpp, , MIT, with more than 127 thousand stars — and got its own website, the llama.app.

GGUF: the format that makes the whole thing work

llama.cpp runs models in the format GGUF, and understanding this saves a lot of confusion.

A GGUF is a single file that packages everything the model needs: the weights, the tokenizer, and the metadata. There's no directory with a dozen files, there's no config.json separate, no surprise of “missing tokenizer”. You download one file and it runs.

And it loads the weights quantized — compressed from 16 bits to 8, 5, 4 or fewer per parameter. It's this compression that allows a 8 billion parameter model to fit in 5 GB instead of 16, and that's why it runs on your notebook.

llama.cpp, Ollama or vLLM?

The most common confusion is thinking they're competitors. They're not exactly:

	llama.cpp	Ollama	vLLM
What is	The engine	A convenience layer	Server engine
Written in	C++	Go	Python
Hardware	CPU, any GPU, Raspberry Pi	CPU and GPU	Dedicated GPU
Format	GGUF	GGUF, via its own registry	Hugging Face weights
Control	Total, down to the finest details	Ready-made patterns	Focus on throughput
Throughput under load	Modest	Modest	It is the project's objective

In summary: the Ollama is easier and uses a good part of this base; the vLLM serves many people simultaneously on a big GPU; llama.cpp is for those who want control, wants to run on modest or exotic hardware, or wants to understand what's really happening.

Installing

The recommended path today is the official installer, which detects the platform and downloads the correct binary:

curl -LsSf https://llama.app/install.sh | sh

llama cli --version

If you prefer to compile — which is what you'll want to do to take advantage of your NVIDIA GPU:

sudo apt update
sudo apt install -y build-essential cmake git libcurl4-openssl-dev

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# CPU apenas
cmake -B build
cmake --build build --config Release -j$(nproc)

# com CUDA (precisa do CUDA Toolkit instalado)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

There are backends for just about everything: -DGGML_CUDA=ON for NVIDIA, -DGGML_HIP=ON for AMD with ROCm, -DGGML_VULKAN=ON for any GPU with Vulkan, -DGGML_METAL=ON on Apple Silicon. The same code, the same models, hand-tuned kernels for each.

First steps: chatting in the terminal

The llama cli It downloads the model from Hugging Face and opens the chat right away. No need to download anything beforehand:

llama cli -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M

The syntax is usuario/repositorio-GGUF:QUANTIZACAO. The model goes into the default Hugging Face cache, which is shared with other tools — meaning it won't duplicate if you already use something else.

# listar o que já está no cache
llama cli -cl

# usar um arquivo .gguf que você já tem em disco
llama cli -m ./meu-modelo.gguf

# uma pergunta só, sem entrar no chat — útil para script
llama cli -m ./meu-modelo.gguf -p "explique o comando tar czf" --no-cnv

At the end of each response it shows the speed in tokens per second, separating prompt processing from generation. That's the metric you want to look at when comparing models, quantization, and tuning.

llama serve: OpenAI-compatible API and web interface

Here the project goes beyond what many people imagine. One command spins up a server with browser-based chat interface is a API compatible with OpenAI:

llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

Open http://localhost:8080 and the interface is already there — no frontend to install. And any application that talks to the OpenAI API can point here by just changing the URL:

curl http://localhost:8080/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "messages": [{"role": "user", "content": "Escreva uma unit systemd para /opt/job.sh"}]
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="nao-importa")

r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Explique inodes em duas frases."}],
)
print(r.choices[0].message.content)

Choosing the right quantization

It's the decision that most affects the result, and the nomenclature is needlessly scary. The practical rule:

Suffix	Bits/weight	When to use
Q8_0	~8	Almost identical to the original; use if you have memory to spare
Q6_K	~6,5	Negligible loss; great if it fits
Q5_K_M	~5,5	Excellent balance
Q4_K_M	~4,8	The sensible default. Start here
Q4_0	~4,5	Faster, slightly worse than K_M
Q3_K_M	~3,9	Only if memory is very tight
Q2_K	~3	Quality drops noticeably; avoid it

Quick memory calculation: multiply the parameter billions by 0,6 GB to get the approximate size in Q4, and add padding for context. An 8B model in Q4_K_M takes about 5 GB.

And the time-saving tip: a larger model with worse quantization usually performs better than a smaller model with better quantization. Between a 14B in Q4 and a 7B in Q8 occupying the same memory, the 14B usually wins.

The parameters that matter

llama serve -m ./modelo.gguf 
  -ngl 99 
  -c 8192 
  --host 0.0.0.0 --port 8080 
  -t 8

• -ngl (n-gpu-layers) — the most important of all. How many model layers go to the GPU. Use -ngl 99 to send everything that fits. If the model doesn't fit entirely, llama.cpp divides between GPU and CPU by itself — and that's exactly what vLLM doesn't do gracefully.
• -c — context size. The larger, the more memory. It's the first parameter to reduce when VRAM is low.
• -t — number of CPU threads. The default usually works; adjust if running on CPU only.
• --host 0.0.0.0 — exposes on the network. Attention: the server has no authentication by default. Use --api-key and a reverse proxy in front if exposing.

To find the best settings for your machine, the project includes a built-in benchmark:

llama-bench -m ./modelo.gguf -ngl 0,20,99

Running as a service

On server, create /etc/systemd/system/llama.service:

[Unit]
Description=llama.cpp server
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=llama
Group=llama
WorkingDirectory=/opt/llama
Environment="LLAMA_CACHE=/opt/llama/models"
ExecStart=/usr/local/bin/llama serve -m /opt/llama/models/modelo.gguf 
  -ngl 99 -c 8192 --host 127.0.0.1 --port 8080
Restart=always
RestartSec=5
TimeoutStartSec=300

[Install]
WantedBy=multi-user.target

sudo useradd -r -s /usr/sbin/nologin -d /opt/llama llama
sudo mkdir -p /opt/llama/models && sudo chown -R llama:llama /opt/llama
sudo systemctl daemon-reload
sudo systemctl enable --now llama
journalctl -u llama -f

The TimeoutStartSec high is intentional: loading a large model from disk takes time, and the default of 90 seconds would kill the service midway.

Quantizing a model yourself

If the model you want doesn't have a GGUF ready, you can convert it:

cd llama.cpp
pip install -r requirements.txt

# baixe os pesos originais do Hugging Face e converta para GGUF em 16 bits
python convert_hf_to_gguf.py /caminho/do/modelo --outfile modelo-f16.gguf

# depois quantize
./build/bin/llama-quantize modelo-f16.gguf modelo-Q4_K_M.gguf Q4_K_M

In practice, it's almost never needed: the community publishes GGUF of everything that matters on Hugging Face just hours after release.

Timeline

• September 2022 — Georgi Gerganov starts GGML, a C tensor library.
• 10 March 2023 — llama.cpp comes out, days after the LLaMA weights leak. LLM running on a laptop, no GPU.
• June 2023 — ggml.ai is founded in Sofia, with investment from Nat Friedman and Daniel Gross.
• 21 of August 2023 — the format GGUF replaces GGML and becomes the de facto standard.
• February 20 of 2026 — Gerganov and the GGML team join Hugging Face, maintaining the technical leadership of the project.
• July 2026 — the llama-cli becomes a thin client of llama-server; the unified CLI arrives llama cli / llama serve and the website llama.app.
• 2 of September 2026 — a NVIDIA announces the acquisition of Hugging Face for US$ 12,93 billion, with closing expected in the first half of 2027, subject to regulatory approval.

What remains — and the irony of the end

The llama.cpp is proof that a well-made engineering decision is worth more than budget. One developer, pure C++, no dependencies, betting that it was possible to run a large model on small hardware — and the entire industry ended up building on top of that.

And here's the irony worth noting: the project that was born to run AI without depending on expensive GPU ended up, in three jumps, inside the NVIDIA — via Hugging Face, if the purchase is approved. Jensen Huang said the platform remains open and that NVIDIA hardware won't be required to use it. The llama.cpp license is MIT, the code is published and a fork is always possible. Still, it's the kind of movement worth watching closely.

To continue: Ollama if you want the easy way, vLLM if you need to serve many people, llama-model to manage your GGUF the refurbished AI server if the idea is to build the machine.

Official links: llama.app · documentation · recommended models · github.com/ggml-org/llama.cpp