After the first server there is often a second, and with it the temptation to join them into a cluster. Proxmox VE makes this possible in a few clicks: one interface for all nodes, migration of virtual machines from host to host, shared configuration. A cluster, however, introduces a concept unknown to anyone coming from a single server: quorum. Understanding it avoids the most common surprise, a node that stops working because the other one is switched off.
What a cluster shares
In a Proxmox cluster the nodes share:
- the configuration of virtual machines, storage and users, replicated by a cluster file system;
- communication through Corosync, a system that keeps track of which nodes are present;
- the ability to migrate virtual machines between nodes.
They do not automatically share disks. If a VM lives on a node's local storage, moving it means copying its disk; with shared storage, such as a NAS over NFS or Ceph, only the memory state needs transferring.
Quorum, simply explained
Each node has a vote. The cluster is operational, or «quorate», only if the nodes that can reach each other hold an absolute majority of the votes. The purpose is to avoid split brain: two halves of the cluster that cannot see each other both believe they are the only active part and change the configuration in diverging ways, or start the same virtual machine twice.
With three nodes the majority is two: if one node goes down, the other two carry on. With two nodes the majority is still two: if one goes down, the survivor has one vote out of two, has no quorum and puts the configuration into read-only mode. Machines already running keep running, but you cannot start, change or create other VMs.
| Nodes | Votes needed | Failures tolerated |
|---|---|---|
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
The table shows why clusters are built with an odd number of votes: the fourth node adds no tolerance over the third.
QDevice: the third vote without a third server
For a homelab with two servers there is an elegant solution: the QDevice. It is a small service, corosync-qnetd, installed on any third machine, even a single-board computer or a virtual machine on a device outside the cluster. It runs no virtual machines: it only provides an extra vote.
With two nodes and a QDevice there are three votes, and losing one node leaves the other with the majority. Setup involves installing the package on the external device and the client on the nodes, then running a configuration command from the cluster.
The QDevice must not run on one of the two nodes: if that node goes down, two votes are lost at once.
The Corosync network
Corosync is sensitive to latency, not bandwidth. If packets arrive late, because a migration or backup saturates the same network, nodes may believe each other gone. Good practice:
- give Corosync a separate network, even just a 1 gigabit port and a small switch;
- configure a second link as a fallback, which Corosync uses if the first fails;
- keep storage and migration traffic off the Corosync network.
High availability: what it really promises
With the HA manager enabled, Proxmox automatically restarts on another node the virtual machines of a node that stops responding. To do this safely, the isolated node must first stop itself: this is called fencing, and Proxmox implements it with a watchdog that reboots a node that has lost quorum.
Conditions for it to work:
- at least three votes, so three nodes or two plus a QDevice;
- VM disks on shared or replicated storage; ZFS replication between nodes is a workable option at home, but in a failure you lose changes made after the last replication;
- accepting that the VM is restarted, not moved live: anyone connected loses their session.
In my work I advise assessing whether HA is really needed. For many home services, a manual restore from backup within a few minutes is a better trade-off than the added complexity.
Mistakes to avoid
- Renaming a node after adding it to the cluster: it is a delicate operation, so choose the name carefully beforehand.
- Adding a node that already has virtual machines: Proxmox advises against it, because IDs may conflict.
- Forcing quorum with
pvecm expected 1without understanding why it was lost: it is an emergency tool, not a permanent fix. - Removing a node from the cluster without following the documented procedure: it leaves inconsistent configurations.
In short
- A cluster unifies management and migration, but does not share disks on its own.
- Quorum needs an absolute majority of votes: with only two nodes, one failure blocks changes.
- A QDevice on a third, external machine provides the missing vote.
- Corosync needs a low-latency network, separate from heavy traffic.
- High availability restarts VMs elsewhere and needs shared or replicated storage: assess it realistically.
