Three Ubuntu machines, one Galera cluster, SST with mariabackup. It is not asynchronous replication: every node accepts writes and the majority decides who stays online. Two nodes are not enough — if one goes down, the other is left alone, with no majority. Below is the three-node cluster from the packa galera_new_cluster, on Ubuntu 24.04 and on 26.04, with hot backup and recovery for when everything goes down.
Three nodes, one quorum
Galera, from the Finnish company Codership, enters MariaDB as the library libgalera_smm.so (package galera-4, wsrep API 26.4.x). Each commit becomes a writeset: the other nodes receive it and certify it before the commit returns to the client, and apply it right after. InnoDB only; MyISAM in the cluster is a headache.
When a node joins or returns, it needs the data. If it was offline only briefly, it receives only what it missed (IST, starting from the cache gcache of a neighbor). If it is new or has been away for a long time, it receives the entire base (SST). The SST via rsync locks the donor during the copy; the mysqldump is slow. The mariabackup copies while hot, sends the stream via socat on port 4444 and the donor keeps serving. That's why three servers and this SST.
The IPs in this how-to — replace them with yours:
db1 10.0.0.11
db2 10.0.0.12
db3 10.0.0.13
Same version of MariaDB on all three, same wsrep_cluster_name, clock synchronized (chrony) and local disk — datadir on NFS, no.
The landscape in September 2026
- Ubuntu 24.04 (noble): MariaDB 10.11.14 in
noble-updatesandgalera-426.4.16 inuniverse. In 2026 May, MariaDB released Galera security patches (10.6.27, 10.11.18, 11.4.12 and 11.8.8). As of the publication of this post, the noble update for 10.11.18 had not yet been released (LP #2158958). For production clusters on 24.04, use the MariaDB repository. - Ubuntu 26.04 (resolute): MariaDB 11.8.6 and
galera-426.4.25, both onmain. The package11.8.6-5ubuntu0.1already includes the May fixes. 11.8 is LTS, with community support until June 2028. - The MariaDB repository also publishes packages for 26.04, but only from 11.8 onward; the installation script refuses 10.11 on resolute.
- Backup binary:
mariadb-backup. The value ofwsrep_sst_methodremainsmariabackup— the old name. Do not mix them.
For those putting PHP in front of the database: Nginx and various PHP versions. The GLPI is one of this cluster's typical clients.
Packages
On all three nodes. Ubuntu 26.04, straight from the distro:
sudo apt update
sudo apt install -y mariadb-server mariadb-backup galera-4 socat
Ubuntu 24.04 with 11.8 from the MariaDB repository (recommended, due to the security patch):
sudo apt install -y curl ca-certificates
curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup \
| sudo bash -s -- --mariadb-server-version="11.8"
sudo apt update
sudo apt install -y mariadb-server mariadb-backup galera-4 socat
The library is in /usr/lib/libgalera_smm.so, with a link in /usr/lib/galera/libgalera_smm.so, both in the Ubuntu package and in the MariaDB one. Either one works on wsrep_provider. Check on your machine:
dpkg -L galera-4 | grep libgalera_smm.so
The installation already left a MariaDB running on its own. Stop it before configuring:
sudo systemctl stop mariadb
sudo systemctl enable mariadb
The datadir changed in 26.04: a new installation uses /var/lib/mariadb; those who upgraded from a previous version remain at /var/lib/mysql (via 99-legacy-datadir.cnf). In 24.04 it is /var/lib/mysql. Check with mariadbd --help --verbose | grep ^datadir.
Network, firewall and AppArmor
Galera uses four ports. Open them only between the three IPs, never to the world:
3306/tcp— SQL client4567/tcp— replication (and UDP, only if you use multicast)4568/tcp— IST4444/tcp— mariabackup/socat SST
NET=10.0.0.0/24
for p in 3306 4567 4568 4444; do
sudo ufw allow proto tcp from "$NET" to any port "$p"
done
sudo ufw reload
On /etc/hosts on all three:
10.0.0.11 db1
10.0.0.12 db2
10.0.0.13 db3
AppArmor on 26.04. The package started shipping the profile /etc/apparmor.d/mariadbd in mode enforce, and it has no rules for the SST scripts (wsrep_sst_mariabackup, socat, mariadb-backup). If the joiner fails and the dmesg shows apparmor="DENIED", set the profile to complaint mode:
sudo apt install -y apparmor-utils
sudo aa-complain /etc/apparmor.d/mariadbd
sudo dmesg -T | grep -i apparmor | tail
In 24.04 the package profile comes empty, without confinement; nothing to do.
The systemd timeout doesn't get in the way: the mariadb.service already ships TimeoutStartSec=900, and during the SST the server itself notifies systemd every 60 s to extend the deadline. No drop-in needed.
60-galera.cnf
Ubuntu already includes /etc/mysql/mariadb.conf.d/60-galera.cnf, all commented out. Replace the contents with this one. Between the nodes only the wsrep_node_name and wsrep_node_address.
In db1:
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="linuxpro"
wsrep_cluster_address="gcomm://10.0.0.11,10.0.0.12,10.0.0.13"
wsrep_node_name=db1
wsrep_node_address=10.0.0.11
wsrep_sst_method=mariabackup
wsrep_sst_auth=mariabackup:troque-esta-senha
wsrep_provider_options="gcache.size=1G;gcache.recover=yes"
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
In db2 and db3, copy the file and change only the name and the IP. The gcomm:// lists all three — Galera ignores its own address. Never leave gcomm:// empty in the file: this creates a new cluster on every start.
The gcache.size defines how much history each node keeps for IST. The higher it is, the longer a node can stay offline and still come back with only what it lost, instead of a full SST.
On 50-server.cnf, comment out bind-address = 127.0.0.1. The 60 is read later and wins, but don't rely on order.
The SST user
The donor runs mariadb-backup locally and needs a user with these privileges:
CREATE USER 'mariabackup'@'localhost' IDENTIFIED BY 'troque-esta-senha';
GRANT RELOAD, PROCESS, LOCK TABLES, BINLOG MONITOR ON *.* TO 'mariabackup'@'localhost';
BINLOG MONITOR is the name since 10.5; REPLICATION CLIENT remains valid as an alias. The user only connects on localhost — don't expose it on the network.
At 11.8 (26.04, or 24.04 with MariaDB's repository) there is a shortcut: without wsrep_sst_auth in cnf, the server automatically creates and deletes a temporary user for each SST (MDEV-31809, since 11.6). In that case, delete the line wsrep_sst_auth from the three files and skip the CREATE USER. On 10.11 the user is mandatory.
Bootstrap and the two joiners
All three are stopped, with the cnf on disk. Only the first node creates the cluster:
# em db1
sudo galera_new_cluster
It's a wrapper that brings up the mariadbd with --wsrep-new-cluster. Check:
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
SHOW GLOBAL STATUS LIKE 'wsrep_ready';
Expected: size 1, Synced, ON. Now create the SST user on db1 (previous section) — it replicates to the others when they join. Then:
# em db2; quando ele estiver Synced, em db3
sudo systemctl start mariadb
The joiner requests the SST from the donor. Monitor with journalctl -u mariadb -f. During the transfer, the datadir receives mariabackup.backup.log (on the donor) and mariabackup.prepare.log / mariabackup.move.log (on the joiner) — that's where the reason is when the SST fails. The files keep the old name, although the binary is mariadb-backup, and those from previous transfers go compressed to /tmp/sst_log_archive. In this post's lab, with the small database, each joiner finished Synced in about 20 seconds.
With all three in:
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; -- 3
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; -- Primary
SHOW GLOBAL STATUS LIKE 'wsrep_incoming_addresses';
Real test: CREATE DATABASE teste_galera; on db1 and SHOW DATABASES; on db3. If it shows up, the writeset got through.
HAProxy in front of the cluster
With three nodes, the application needs a single address that keeps working when one node goes down. HAProxy does this over TCP. And there is a reason not to spread the writes: Galera accepts writes on any node, but two transactions touching the same row on different nodes cause a certification conflict, and one of them comes back with a deadlock error in COMMIT. The safest arrangement is write to one node at a time and read from all three.
HAProxy exposes two ports: 3306 for writes, which sends everything to db1 and uses db2 and db3 only if it goes down, and 3307 for reads, distributed across all three. If the application doesn't separate reads from writes, use only 3306.
Health check: TCP is not enough
A node in Donor/Desynced, still syncing or outside the quorum accepts TCP connections normally — it just doesn't serve reliable data. The check needs to ask Galera. This script returns HTTP 200 when wsrep_local_state is 4 (Synced) and 503 in any other case. On all three nodes:
sudo tee /usr/local/bin/galera-check >/dev/null <<'EOF'
#!/bin/bash
# Responde HTTP 200 só quando o nó está Synced (wsrep_local_state = 4).
# Lê o pedido até a linha em branco; fechar sem ler faz o kernel mandar RST.
while read -r -t 2 line; do [ -z "${line%$'\r'}" ] && break; done
state=$(mariadb -N -s -e "SHOW GLOBAL STATUS LIKE 'wsrep_local_state'" 2>/dev/null | cut -f2)
if [ "$state" = "4" ]; then
printf 'HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\nSynced\r\n'
else
printf 'HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\nNot synced (%s)\r\n' "${state:-down}"
fi
EOF
sudo chmod 755 /usr/local/bin/galera-check
sudo useradd -r -s /usr/sbin/nologin galera-check
The script connects without a password: the MariaDB user authenticates via the system user (unix_socket) and doesn't need any privileges to read the status. Create it once, on any node — it replicates:
CREATE USER 'galera-check'@'localhost' IDENTIFIED VIA unix_socket;
Systemd listens on port 9200 and runs the script on every connection. On all three nodes, /etc/systemd/system/galera-check.socket:
[Unit]
Description=Health check HTTP do Galera (porta 9200)
[Socket]
ListenStream=9200
Accept=yes
[Install]
WantedBy=sockets.target
and /etc/systemd/system/galera-check@.service:
[Unit]
Description=Health check HTTP do Galera
[Service]
User=galera-check
ExecStart=/usr/local/bin/galera-check
StandardInput=socket
StandardOutput=socket
StandardError=journal
sudo systemctl daemon-reload
sudo systemctl enable --now galera-check.socket
curl -s http://127.0.0.1:9200/ # Synced
Open 9200 only for the HAProxy IPs (sudo ufw allow proto tcp from 10.0.0.21 to any port 9200, and the same for .22).
The haproxy.cfg
On the HAProxy machines (outside the database nodes):
sudo apt install -y haproxy
Add to /etc/haproxy/haproxy.cfg, keeping the block global of the package:
defaults galera
mode tcp
log global
option tcplog
timeout connect 5s
timeout client 8h
timeout server 8h
# Escrita: um nó só por vez; os outros dois assumem se ele cair
frontend galera_escrita from galera
bind *:3306
default_backend galera_escrita
backend galera_escrita from galera
option httpchk GET /
http-check expect status 200
default-server port 9200 inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
server db1 10.0.0.11:3306 check
server db2 10.0.0.12:3306 check backup
server db3 10.0.0.13:3306 check backup
# Leitura: os três, quem tiver menos conexões
frontend galera_leitura from galera
bind *:3307
default_backend galera_leitura
backend galera_leitura from galera
balance leastconn
option httpchk GET /
http-check expect status 200
default-server port 9200 inter 2s fall 3 rise 2 on-marked-down shutdown-sessions
server db1 10.0.0.11:3306 check
server db2 10.0.0.12:3306 check
server db3 10.0.0.13:3306 check
listen estatisticas from galera
mode http
bind 127.0.0.1:8404
stats enable
stats uri /
stats refresh 5s
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
The details that matter:
fall 3withinter 2s: a node leaves the pool after three failed checks, roughly 6 seconds.rise 2it requires two good ones to come back.on-marked-down shutdown-sessionsit drops the open connections on the failed node, so the application reconnects to the next one instead of getting stuck.timeout client/server 8hit follows thewait_timeoutMariaDB standard (28800 s). With a smaller timeout, HAProxy drops idle pool connections before the database does.- Application users will now arrive with the HAProxy IP: create them as
'app'@'10.0.0.%', not with the application IP.
Test through the load balancer:
mariadb -h 10.0.0.21 -P 3306 -u app -p -e 'SELECT @@hostname' # sempre db1
mariadb -h 10.0.0.21 -P 3307 -u app -p -e 'SELECT @@hostname' # alterna entre os três
In the lab from this post, with db1 down, writes went to db2 in less than 10 seconds, and one INSERT on port 3306 came in normally. When db1 came back (via IST, only with the transactions it lost), HAProxy returned writes to it. To see the state of the backends: ssh -L 8404:127.0.0.1:8404 on the HAProxy machine and open http://127.0.0.1:8404/.
A single HAProxy becomes the new single point of failure. In production, run two of them, with the same configuration, and keepalived holding a virtual IP (VRRP) between them, as in the diagram; the application points to that IP.
Hot backup
A cluster backup is a backup of a node. Ideally remove this node from the load balancer, because it becomes slower during the copy:
sudo mkdir -p /backup # o mariadb-backup não cria o diretório pai
sudo mariadb-backup --backup --galera-info \
--target-dir=/backup/full-$(date +%F) \
--user=mariabackup --password='troque-esta-senha'
sudo mariadb-backup --prepare --target-dir=/backup/full-$(date +%F)
No need for SET GLOBAL wsrep_desync=ON: in Galera, the mariadb-backup already desynchronizes and pauses the node on its own during the blocking phase. The node remains in the cluster, receives the writesets in a queue and drains it later — there is no IST. The --galera-info records the cluster position in the file xtrabackup_galera_info, which is what allows this backup to be used to rebuild a node.
The --prepare keeps the backup consistent. Without it, the directory cannot be used for restoration.
When the entire cluster goes down
If three fall together, the cluster is gone — the data is not. On each node, check the grastate.dat in the datadir:
sudo cat /var/lib/mysql/grastate.dat # ou /var/lib/mariadb no 26.04
Clean shutdown: the node with safe_to_bootstrap: 1 (the last to leave) is the one that comes up first. Sudden crash: all of them remain with seqno: -1 and safe_to_bootstrap: 0. Find out the actual position on each one:
sudo galera_recovery
The node with the highest number after : is the most advanced. On that one only, edit the grastate.dat to safe_to_bootstrap: 1 and:
sudo galera_new_cluster # no nó mais avançado
sudo systemctl start mariadb # nos outros dois, um de cada vez
Never run galera_new_cluster with any node still up: a new cluster is born with a different UUID and you end up with two databases diverging (split-brain).
Upgrading the major version
Those on 10.11 of 24.04 moving to 11.8: SST via mariabackup does not work between some major versions, because the redo log format changed in 10.5 and in 10.8 (MDEV-27437). During migration use wsrep_sst_method=rsync, which crosses versions, or stop all three, upgrade everyone, and bootstrap again.
What usually breaks
wsrep_sst_method=mariadb-backup— wrong name. It ismariabackup; there is nowsrep_sst_mariadb-backup.- AppArmor script in 26.04 denying the SST; the MariaDB log only says “SST failed”. Check the
dmesg. socatmissing on the donor or the joiner.- Port 4444 closed in the firewall: the joiner waits and the SST dies by timeout.
- Two nodes: one goes down and the other becomes
Non-Primaryand stops accepting writes. Use three, or two plus an arbitergarbd. bind-address=127.0.0.1still active: the others don't connect.- MyISAM table or without primary key: replicates poorly or doesn't replicate at all. Convert to InnoDB and create the PK.
References: provider code in github.com/codership/galera · Getting Started · SST with mariadb-backup · Configuring MariaDB Galera Cluster · DEB packages.