So in my last post about my Proxmox home lab I said I was clustering the Proxmox nodes. Wasn’t horrible but I did seem to find CLI could make things happen easier than some of the GUI menus.
Three nodes. One PC and a couple of NUC’s. Ill call them pve01 (the PC), pve02 and pve03 (the NUC’s), and the cluster is called homelab.
Why cluster at all?
Honestly? One GUI. Instead of three browser tabs and three logins, I log into any node and see (and manage) everything. Moving a VM from one box to another becomes a right-click instead of a backup-copy-restore dance.
What MY cluster is NOT; automatic failover. HA in Proxmox wants shared storage, and my VMs live on local disks. So if a node dies, the VMs on it are down until I restore them somewhere else. Set your expectations accordingly. That may change in the future – but for now – this is good enough.
Cleaning existing VM’s
A node has to be empty to join. The node that creates the cluster keeps its VMs, but every node that joins can’t have a single VM or container on it. Its config folder (/etc/pve) gets replaced with the cluster’s copy. Two of my nodes had VMs, so the plan was: back up, verify, delete, join, restore.
And yes – verify. If you don’t care about rebuilding – then don’t verify – but if it could be a pain – then do this so you know the backup is correct – and hash values help validate files.
# sha256sum /var/lib/vz/dump/*.zst
# zstd -t /var/lib/vz/dump/vzdump-qemu-102-*.vma.zst
VM IDs will collide. My desktop and one of the NUC’s both had VMs numbered 100 through 103. Inside a cluster every ID has to be unique, so when I restored, the NUC’s 102 became 202. Lesson learned: give each node its own ID range from day one (100s on one, 200s on another, etc.) and you’ll never have to think about it.
Pick your hostnames now. Renaming a node after it joins is a PITA. Mine kinda suck, but they’re unique, and that’s what matters.
Same Proxmox version everywhere. Check every node before joining, and update the stragglers:
# apt update && apt full-upgrade
# pveversion - Double check versions are all on the same buildJoining
On the desktop:
# pvecm create homelab
# pvecm status
Then on each NUC, either Datacenter > Cluster > Join Cluster in the GUI (paste the Join Information from the first node), or:
# pvecm add X.X.X.X
It asks for the first node’s root password and a fingerprint. The joining node’s web page will probably drop or throw a certificate warning for a minute – that’s normal, it’s switching over to the cluster’s certificates. SSH will also complain that the host key changed. Also normal:
# ssh-keygen -R X.X.X.X
The first NUC joined without drama. The second one… not so much.
The node that stood alone
pve03 joined, and then the GUI wouldn’t load and I couldn’t log in. Um.. what? This is where the CLI earned its keep:
# pvecm status
# pvecm nodes
On pve03 it showed exactly one member – itself – and this:
Quorum: 2 Activity blocked
A cluster needs more than half the nodes voting (quorum) before it’ll let anything change. pve03 could only see itself, so it locked everything down, including logins. The other two still had 2 of 3 votes, so they were fine.
The logs told the rest of the story:
# journalctl -u corosync -u pve-cluster -n 50
# cat /etc/pve/corosync.conf
FW rules might get you – Corosync on pve03 was trying to form a cluster about every 7 seconds and getting nothing back from the other two. Corosync is the service that carries cluster traffic, and my nodes aren’t all on the same subnet. A firewall rule between the subnets was blocking the cluster traffic. Corosync needs UDP 5405-5412 open between all nodes, plus TCP 22 (SSH) and 8006 (GUI). I added the rule, and:
Quorate: Yes
Success 🙂
Storage is shared now (surprise)
Once you cluster, there’s one storage list for every node. There were a couple of things that popped up.
The joining node’s storage config gets replaced. pve03 had the NAS backup storage defined, and after joining… gone (the files on the NAS were fine, only the config changed). Good news: you add it once from any node and all three get it. I ended up with one folder per node on the NAS, each added as its own storage and visible to every node:
# pvesm add cifs nas-pve03 --server X.X.X.X --share lab-backups --subdir /pve03/backup --username youruser --password --content backup --smbversion 2.0
# pvesm status
Note – Proxmox only lists backups sitting inside a dump folder under that subdir. If your files are loose, move them into dump/ and they’ll show up.
Local storage shows up everywhere too. My desktop has a “backups” storage on a disk only the desktop has. After clustering, the other two nodes showed it with an error, because they went looking for a disk they don’t have. The fix is telling Proxmox which node it lives on:
# pvesm set backups --nodes pve01
Errors gone.
Moving VMs around
This was the whole point, right? Migrate is right there in the right-click menu. And for VMs on matching storage it just works.
Mine didn’t all have matching storage. My nodes use different local storage names, and the GUI’s offline migration didn’t want to play along. The CLI let me pick the target storage:
# qm migrate 105 pve02 --targetstorage local-ssd
Shut the VM down first. Offline migration is way less likely to surprise you.
Migrated VMs bring their old network settings with them. One VM came up on its new node and wouldn’t ping anything. The VM was fine – its network config still matched the old node’s network. Once I fixed the NIC settings for where it lives now, it was back. Check the bridge and IP after every move.
Next on the list is Proxmox Backup Server… and the bad RAM story I promised. That one gets its own post (currently still in progress).
Hope this helps someone searching Google for the same problems. Happy labbing!
