Running local AI on Linux with Ollama

Running AI models locally is no longer a lab curiosity: with Ollama you download and run LLMs on your own Linux machine, without sending data to the cloud and without paying per token. This post explains where the project came from — the origin explains a lot about its design — and shows how to install, configure, and actually use it.

Where Ollama came from

To understand why Ollama feels so much like Docker, just look at who wrote it. Jeff Morgan and Michael Chiang are two Canadians who met at the University of Waterloo and, before all this, created Kitematic — the interface that made Docker usable by ordinary people. Docker bought the project in 2015, and that became what is now called Docker Desktop.

The duo went through Y Combinator in the winter 2021 class and, on 26 June 2023, opened Ollama's repository. The thesis was the same as Kitematic, swapping containers for models: there is a powerful technology that is annoying to operate, and someone needs to wrap it in a single command.

That is where everything familiar comes from. You pull a model from a registry. You describe your model in a file called Modelfile, which is a Dockerfile by another name. You list what is installed with list and delete it with rm. None of this is a coincidence.

Figures for September 2026: written in Go, licensed under MIT, more than 180 thousand stars on GitHub and around 9 million developers using it. In July 2026 the company — with just 14 people — raised a Series B of US$ 65 million led by Theory Ventures, reaching US$ 88 million in total, with Benchmark and Y Combinator itself on the list.

An important technical detail in its trajectory: until 2025 Ollama was essentially a wrapper around llama.cpp. In May that year the project launched its own engine, built on the GGML tensor library, because supporting multimodal models by patching llama.cpp had become unsustainable. Today each model is self-contained and brings its own projection layer — but GGML is still underneath, for CPU compatibility.

Why run local AI?

• Privacy: your data does not leave the machine — decisive for proprietary code, internal documents, and anything covered by Brazil's LGPD.
• Zero per-use cost: after downloading the model, run it as many times as you want.
• Works offline: useful on restricted networks, isolated labs, and servers without internet egress.
• No rate limits: no “you have exceeded your quota” in the middle of batch processing.

And the other side, which is only fair to mention: the quality is not the same. An 8B model running on your GPU does not replace a frontier model. For mechanical tasks — summarizing, classifying, extracting, reformatting — the difference disappears. For difficult reasoning, it does not.

Real requirements: it's memory that matters

The practical calculation: a model quantized to 4 bits occupies roughly 0,6 GB per billion parameters, plus some headroom for context. In other words:

Model size	VRAM or RAM required	Useful for
1B – 3B	~2–4 GB	Classification, extraction, autocomplete; runs even on a CPU
7B – 8B	~6–8 GB	The best value for general use
14B	~10–12 GB	Code and reasoning, with a card of 12 GB or more
32B	~20–24 GB	Needs a large card; much better quality
70B+	40 GB or more	Two GPUs or a lot of patience with RAM offloading

If the model does not fit in VRAM, Ollama puts the excess in system RAM and keeps working — just slowly. If you want to know where to invest, see the refurbished AI server with 64 GB of VRAM and the 32 GB Tesla V100.

Installing

One-line installation, straight from the official website:

curl -fsSL https://ollama.com/install.sh | sh

The installer creates the systemd service ollama and detects the GPU automatically. Check:

systemctl status ollama
ollama --version
nvidia-smi        # a GPU foi vista?

If you prefer a container — useful for isolating the installation or running on an already containerized server:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 
  --name ollama ollama/ollama

docker exec -it ollama ollama run qwen2.5-coder

Downloading and running a model

# baixa e já abre o chat interativo
ollama run llama3.2

# só baixar, sem abrir o chat
ollama pull qwen2.5-coder

# tamanho específico
ollama pull llama3.2:3b

# listar o que está instalado, com tamanho
ollama list

# ver o que está carregado na memória agora
ollama ps

# uma pergunta só, direto do shell — ótimo para script
ollama run llama3.2 "explique o comando: tar czf backup.tar.gz /etc"

# ler um arquivo e mandar para o modelo
ollama run llama3.2 "resuma os erros deste log" < /var/log/syslog

Inside the interactive chat, /bye exits, /clear forgets the conversation, and /set verbose starts showing speed in tokens per second — the metric you want to watch when comparing models.

For a modest machine, prefer smaller variants. For programming, the coder families perform noticeably better than general-purpose models of the same size.

The Modelfile: Docker's legacy

Here the creators' background is on full display. You customize a model by declaring what you want in a text file — the same logic as a Dockerfile:

FROM qwen2.5-coder:7b

# temperatura baixa = respostas mais previsíveis
PARAMETER temperature 0.2
PARAMETER num_ctx 8192

SYSTEM """
Você é um assistente de sysadmin Linux. Responda em português do Brasil,
sempre com o comando pronto para colar. Avise quando o comando for
destrutivo. Não invente flag que você não tem certeza que existe.
"""

ollama create sysadmin -f ./Modelfile
ollama run sysadmin

It is the fastest way to turn a generic model into a tool tailored to your work — and the result is saved as your own local model, which you can list, run, and delete like any other.

Using the API

Ollama starts a local HTTP API on port 11434. The native one:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explique o comando: tar czf backup.tar.gz /etc",
  "stream": false
}'

And — making life easier for those who already have working code — it also speaks the OpenAI protocol. Any application or SDK using the OpenAI API can point to your machine by changing only the URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resposta = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Escreva uma unit systemd para um script em /opt/job.sh"}],
)
print(resposta.choices[0].message.content)

The settings that matter

The default works, but it is almost never what you want on a server. Configuration goes in systemd:

sudo systemctl edit ollama

[Service]
# escutar na rede, não só no localhost
Environment="OLLAMA_HOST=0.0.0.0:11434"

# guardar os modelos num disco maior
Environment="OLLAMA_MODELS=/dados/ollama"

# manter o modelo carregado (padrão: 5 minutos)
Environment="OLLAMA_KEEP_ALIVE=30m"

# quantas requisições atender em paralelo
Environment="OLLAMA_NUM_PARALLEL=4"

# quantos modelos diferentes manter na memória ao mesmo tempo
Environment="OLLAMA_MAX_LOADED_MODELS=2"

sudo systemctl daemon-reload && sudo systemctl restart ollama

Security warning: Ollama has no authentication. By changing OLLAMA_HOST to 0.0.0.0 you open port 11434 to anyone who can reach the machine — and anyone who can reach it can use your GPU and read your conversations. On an exposed server, put a reverse proxy with authentication in front and block 11434 in the firewall.

Ollama, llama.cpp, or vLLM?

	Ollama	llama.cpp	vLLM
Who it is for	Personal use and small teams	Those wanting fine-grained control	Multiuser production services
Ease of use	One command and it works	Requires compiling and tuning	Requires VRAM planning
Format	GGUF, via its own registry	GGUF	Hugging Face weights
Model does not fit	Offloads to RAM	Offloads to RAM	Needs --cpu-offload-gb
Throughput under load	Drops quickly	Drops quickly	It is the project's objective

In short: if it is you on your machine, Ollama. If it is an API that several people or several agents will consume simultaneously, the way to go is vLLM.

Maintenance and cleanup

Models occupy several GB and fall off the radar. From time to time:

ollama list                 # o que eu tenho mesmo?
ollama rm llama3.2          # apagar um
du -sh ~/.ollama/models     # quanto isso está comendo de disco

Models live in ~/.ollama/models when you run as a user, and in /usr/share/ollama/.ollama/models when running as a service.

Timeline

• 2015 — Docker buys Kitematic, by Jeff Morgan and Michael Chiang; the project becomes the foundation of Docker Desktop.
• Winter 2021 — the duo goes through Y Combinator.
• 26 June of 2023 — the Ollama repository is created, in Go and under the MIT license, as a layer over llama.cpp.
• May 2025 — the project launches its own engine on top of the GGML library to handle multimodal models.
• July 2026 — a Series B of US$ 65 million led by Theory Ventures; total funding reaches US$ 88 million, with around 9 million developers using it.
• September 2026 — more than 180 thousand stars on GitHub; the stable branch is 0.33.

Closing

Ollama's merit is not technical — llama.cpp already did local inference before it. The merit is in packaging: two guys who had done exactly that with Docker looked at LLMs and turned a process of compiling, converting, and tuning parameters into ollama run. It is the same lesson as Kitematic, ten years later.

To continue: AI assistants on the Linux terminal shows how to plug this into your workflow, and GPU vs TPU explains what is happening in the hardware while the model responds.

Official links: ollama.com · documentation · github.com/ollama/ollama · releases