A snapshot isn't a backup

A Proxmox snapshot is a point-in-time state stored on the same storage as the VM. It's perfect for "I'm about to upgrade this and want to roll back in five minutes if it goes badly." It does nothing for you if the storage fails, the pool fills, someone deletes the VM, or ransomware reaches the host. Long-lived snapshots also slow down some storage types and eat space as the disk changes. Take them before a change and delete them after. Backups are separate copies on separate storage.

Picking a vzdump mode

• Snapshot mode keeps the guest running while it's backed up. For VMs, Proxmox uses QEMU's live backup mechanism, and with the QEMU guest agent installed and enabled it briefly freezes the guest filesystems for consistency. For containers it needs storage that supports snapshots, or it falls back to suspend mode. This is our default.
• Suspend mode, for containers, copies data while running and then suspends the container briefly for a final sync. For VMs it offers little over snapshot mode and causes a pause.
• Stop mode shuts the guest down, backs it up and starts it again. It's the most consistent, at the cost of downtime. We use it for a few legacy systems whose databases don't tolerate live backup well.

Install and enable the QEMU guest agent on every VM: qemu-guest-agent on Linux, the VirtIO guest tools on Windows, and the Guest Agent option in the VM settings. Without it, snapshot-mode backups are crash-consistent, not filesystem-consistent.

A one-off manual backup looks like this:

vzdump 101 --mode snapshot --storage backup-nfs --compress zstd --notes-template '{{guestname}} manual'

Scheduling, and why hand-picked lists go wrong

Create jobs under Datacenter, Backup in the GUI. They're stored cluster-wide in /etc/pve/jobs.cfg and run on whichever node holds each guest. We usually run one nightly job for everything, outside business hours, selecting by pool or by "all" with exclusions instead of a hand-picked list of VM IDs. Hand-picked lists are how a new VM ends up never backed up.

Point the job's failure notifications at someone who actually reads that inbox. A backup job that's been failing silently for three weeks is the most common problem we find on inherited clusters.

How long to keep things

Retention is set with prune options on the job or on the storage. For most small environments we start here:

keep-last=3, keep-daily=7, keep-weekly=4, keep-monthly=6

On the command line the same thing is:

vzdump 101 --storage backup-nfs --prune-backups keep-daily=7,keep-weekly=4,keep-monthly=6

That gives you a week of dailies, a month of weeklies and half a year of monthlies. The monthlies are what save you when someone notices that data went missing back in the spring.

Proxmox Backup Server is worth the extra box

Backing up to a plain NFS share works, but every backup is a full compressed archive. PBS changes that in a few ways:

• Data is stored in chunks and identical chunks are stored once, so nightly backups of mostly unchanged VMs take a small fraction of the space.
• With dirty bitmaps, a running VM only sends changed blocks after the first backup, so jobs finish much faster.
• Verify jobs re-read chunks and check their checksums on a schedule, which catches silent corruption before you need the backup.
• Sync jobs to a second PBS offsite give you the offsite copy in a 3-2-1 setup.
• You also get file-level restore from the GUI, encryption, and per-user permissions.

On PBS, schedule prune, garbage collection and verify jobs per datastore. Prune only marks old snapshots for removal. Garbage collection is what actually frees the space.

Restore to a new ID and actually boot it

A backup you've never restored is a hope, not a backup. Restore to a new VM ID so the production guest is left alone:

# VM from a vzdump archive on file storage
qmrestore /mnt/pve/backup-nfs/dump/vzdump-qemu-101-2026_10_01-02_00_03.vma.zst 9101 --storage local-lvm --unique

# VM from PBS (volume ID as shown by pvesm list)
pvesm list pbs01 --vmid 101
qmrestore pbs01:backup/vm/101/2026-10-01T09:00:03Z 9101 --storage local-lvm --unique

# Container
pct restore 9201 /mnt/pve/backup-nfs/dump/vzdump-lxc-201-2026_10_01-02_10_44.tar.zst --storage local-lvm

Before you boot the copy, take it off the network. Otherwise it can fight the production guest for its IP or start sending email.

qm set 9101 --net0 virtio,bridge=vmbr0,link_down=1
qm start 9101

When you rewrite net0, keep the NIC model and bridge that qm config 9101 shows. Then open the console and confirm the OS boots, services start and the data is current. Delete the test guest when you're done.

We restore something every month. If the dated note for an important system is more than a month old, that restore is the next thing on my list.

Write down what every box really runs

Before moving anything, log into every VPS and write down what it actually runs, not what you remember it running. For each box we captured:

• Listening ports and the processes behind them (ss -tlnup)
• Every hostname pointing at it, from DNS and from web server configs
• Cron jobs and systemd timers for every user
• Databases, where their data lives, and how big they are
• Inbound dependencies like webhooks, firewall allowlists at third parties, and API clients that connect by IP
• Outbound dependencies like SMTP relays and API keys bound to source IPs

The cron jobs and IP allowlists are what get you. A forgotten nightly report, or a vendor that only accepts connections from your old IP, won't show up until a week after cutover when someone asks where their report went.

One container per service

I get the urge to rebuild the old all-in-one boxes as they were. Resist it. We gave each service its own LXC container (or a VM where it needed one) with its own IP on an internal services VLAN, for example 10.10.20.0/24. Now each service gets backed up, restored, upgraded and moved on its own, and the next migration becomes moving one small thing at a time.

You own the firewall now

Your VPS provider was giving you a firewall for free, even if you never thought about it. Now it's your job. We run a VyOS VM with the public uplink on one side and the internal VLANs on the other. The policy is simple. Inbound is dropped by default, established and related return traffic is allowed, and only the specific ports each published service needs are opened. Internal VLANs can reach the internet, but the internet reaches nothing unless a rule says so.

A single reverse proxy for the web stuff

Most of what we moved was HTTP. Instead of giving every web app a public IP, we run one reverse proxy container and forward ports 80 and 443 to it. Ours is Nginx Proxy Manager, though Caddy, Traefik or plain Nginx are just as good. It terminates TLS, holds the certificates, and routes by hostname to the right internal container, so one public IP can serve dozens of sites.

1:1 NAT for the few that need their own IP

Some things don't fit behind a shared HTTP proxy. Think mail servers that need matching reverse DNS, VPN endpoints, game or voice servers on fixed ports, and anything a third party allowlists by IP. For those we use 1:1 NAT on the router, mapping a dedicated public address to one internal container, with firewall rules limited to the ports that service uses. Keep this list short, because every 1:1 mapping is one more exposed surface to watch.

Test the new side while DNS still points at the old one

This is the step that made our cutovers boring, which is exactly what you want. Copy the data, bring the service up on the new side, and test it while DNS still points at the old server. Lower the DNS TTL to 300 seconds a day or two ahead so the eventual flip propagates quickly.

From inside the network, hit the backend directly with a Host header:

curl -sI -H 'Host: app.contoso.com' http://10.10.20.15/

Then test the full public path, including the router, reverse proxy and TLS certificate, without changing DNS. --resolve pins the hostname to the new public IP:

curl -sv --resolve app.contoso.com:443:192.0.2.10 https://app.contoso.com/ -o /dev/null 2>&1 | grep -E 'HTTP/|subject:|expire'

You can do the same thing in a browser by temporarily adding the name to your hosts file. Log in, click around, submit a form, and check that background jobs run. Only once it works end to end do we touch DNS.

Flip DNS, but keep a way back

For services with a database, do a final sync in a quiet window. Stop the app on the old side, take a last dump, restore it on the new side, then flip DNS. After that:

• Stop the old VPS. Don't delete it. A stopped instance can be started again in a minute if something was missed.
• Take a provider snapshot of the old box as a second rollback and keep it until you're confident. Snapshots usually cost a fraction of a running instance.
• If you can, watch the old box's logs for a few days before stopping it. Any traffic still arriving points you at a hard-coded IP or a forgotten client.

Moving one small service at a time felt slow while we were doing it. Looking back, it's the reason nobody had to call us about it.

Stay unprivileged

In an unprivileged container, root inside the container maps to a high, unprivileged UID on the host (UID 100000 by default). If something breaks out, it lands on the host as a nobody user. In a privileged container, root inside is root on the host, so a container escape is a host compromise.

Privileged mode does avoid some friction, like bind-mounted host directories with odd ownership, certain device access, or NFS mounts inside the container. We still treat it as a last resort. If an app only works in privileged mode, that usually tells me it belongs in a VM.

When I build a VM instead

• The workload needs its own kernel, kernel modules or a different kernel version.
• You need live migration without downtime. Containers restart when they migrate.
• The service is internet-facing and untrusted, and you want the stronger isolation boundary.
• The stack leans on Docker features that fight with nesting, or it's Kubernetes.
• It's Windows, or anything else that isn't Linux.

For a handful of internal services, Docker in LXC is fine, and we run plenty of them.

Creating the container

Two features matter. nesting=1 lets the container create its own nested namespaces and cgroups, which Docker needs. keyctl=1 allows the keyctl system call in unprivileged containers. Docker and runc use it, and without it you get failures that are confusing to track down.

pveam update
pveam available --section system | grep debian-12
pveam download local debian-12-standard_12.7-1_amd64.tar.zst

pct create 210 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname docker01 \
  --unprivileged 1 \
  --features nesting=1,keyctl=1 \
  --cores 2 --memory 2048 --swap 512 \
  --rootfs local-lvm:16 \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.20.21/24,gw=10.10.20.1 \
  --onboot 1

pct start 210

Use the template filename that pveam available actually lists, because the version suffix changes over time. For a container you already have, turn the features on under Options, Features in the GUI, or run pct set 210 --features nesting=1,keyctl=1 and restart it.

Installing Docker

Get in with pct enter 210 and install Docker from Docker's official repository using their documented Debian steps, or use the distribution's docker.io package if you prefer. Then run a test container:

docker run --rm hello-world
docker info | grep -i 'storage driver'

You want the overlay2 storage driver. If you see vfs, every image layer is a full copy and disk use balloons. That used to be common on ZFS-backed container storage. Newer ZFS releases support overlayfs, so check what your host is running.

Giving it /dev/net/tun

Tailscale and WireGuard userspace implementations need /dev/net/tun, and unprivileged containers don't get it by default. Stop the container and add two lines to its config file on the host:

pct stop 210
nano /etc/pve/lxc/210.conf

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

The first line allows access to the character device with major 10, minor 200, which is TUN. The second bind-mounts the host's device node into the container and creates the file if it doesn't exist. Start the container again and install Tailscale the normal way:

pct start 210
pct enter 210
ls -l /dev/net/tun
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up

If the container is going to be a subnet router or exit node, also enable IP forwarding inside it (net.ipv4.ip_forward=1 in a sysctl file) and approve the routes in the Tailscale admin console. That last bit is the part I always forget.

I try to keep it to one service per container. Once a single container is running twenty stacks, you've rebuilt a VM with weaker isolation, and that's usually my cue to move it to an actual VM.

Networks first, because moving Ceph later hurts

Changing Ceph networks after the fact is painful, so we settled them before installing anything. We use four logical networks, each on its own VLAN:

• Management (for example 10.10.10.0/24) carries the Proxmox web UI, SSH and corosync. Corosync is latency-sensitive, so keep it off any link Ceph is saturating. A second corosync link on another network is cheap insurance.
• Ceph public (10.10.30.0/24) is where monitors and clients talk to OSDs.
• Ceph cluster (10.10.40.0/24) carries OSD-to-OSD replication and recovery. This is the link that gets hammered when a disk fails.
• VM traffic rides a VLAN-aware bridge carrying the guest networks.

On a two-port 10GbE node, we put Ceph public and cluster on one port, management and VM traffic on the other, and used the 1GbE ports as a corosync backup.

Disks: Proxmox on the boot device, everything else to Ceph

Nutanix nodes usually boot from a small SATADOM or M.2 device and keep the data drives behind an HBA. Install Proxmox on the boot device and leave every data disk for Ceph. Give each data disk its own OSD and don't put RAID underneath it, since Ceph is your redundancy layer.

Keep SSDs and HDDs in separate pools with separate CRUSH rules. If you mix them in one pool, the slowest disk sets the pace for everything. And set expectations with whoever is planning capacity: with the default 3-way replication, usable space is roughly a third of raw.

Install every node, then ping everything

Install from the ISO on each node with a unique hostname and a static management IP. After first boot, set up your repositories, update, and build the bridges and VLAN interfaces in /etc/network/interfaces or the GUI. Before you go further, make sure every node can ping every other node on the management, Ceph public and Ceph cluster networks. A VLAN typo is a lot easier to find now than after Ceph is running on top of it.

Building the cluster

On the first node:

pvecm create hx-cluster

Then join the rest using the first node's management IP:

pvecm add 10.10.10.11
pvecm status

pvecm status should list every node and show Quorate: Yes. I join nodes one at a time and let each one settle before adding the next.

Ceph packages, monitors and OSDs

Install the Ceph packages on every node, then initialize the config once with both networks:

# on every node
pveceph install --repository no-subscription

# once, on the first node
pveceph init --network 10.10.30.0/24 --cluster-network 10.10.40.0/24

Monitors need a majority to keep quorum, so run an odd number of them. On four nodes that means three. Run these on three different nodes:

pveceph mon create
pveceph mgr create

Then create one OSD per data disk on each node. Ceph assigns the device class (hdd, ssd or nvme) by itself, and you can check it later with ceph osd tree.

lsblk -o NAME,SIZE,ROTA,MODEL
pveceph osd create /dev/sdb
pveceph osd create /dev/sdc
# ...repeat for every data disk on every node

Pools split by disk type

Create one replicated rule per device class, then a pool for each. The --add_storages flag registers the pool as Proxmox storage across the cluster.

ceph osd crush rule create-replicated rule-ssd default host ssd
ceph osd crush rule create-replicated rule-hdd default host hdd

pveceph pool create vm-ssd --crush_rule rule-ssd --add_storages
pveceph pool create vm-hdd --crush_rule rule-hdd --add_storages

Leave the defaults of size=3 and min_size=2. We put VM boot disks and databases on the SSD pool and bulk data on the HDD pool.

Watch the order here. If you create a pool before your class-specific rules exist, it uses the default rule that spans every disk. You can change it later with ceph osd pool set POOL crush_rule rule-ssd, but that triggers data movement, so it's easier to set the rule when you create the pool.

CephFS for the ISO pile

RBD pools hold disk images, but ISOs, container templates and backups-in-transit need a filesystem. Put a metadata server on at least two nodes for failover, then create the filesystem:

pveceph mds create   # on two or more nodes
pveceph fs create --name cephfs --add-storage

Every node now sees the same ISO and template library. It's a small thing, but it's one of my favorite quality-of-life wins over local storage.

ceph -s
ceph osd tree
ceph osd df
pvesm status

A few more things I'd tell the next person

• Update BMC and BIOS firmware while the nodes are still empty, and confirm the HBA presents disks directly to the OS.
• Point chrony at a reliable NTP source on every node. Monitors complain about clock skew if the node clocks drift.
• Don't fill it up. Ceph gets unhappy as OSDs approach the nearfull ratio. We plan to stay well under 80 percent so a node failure can re-replicate without hitting full.

If I were doing it again, I'd spend even longer on the network plan, since that's the one decision here that's genuinely painful to undo.