8 hours ago8 hr When the support contract ran out on a 4-node Nutanix block, the easy move would have been to recycle it. I'm glad we didn't. Those chassis are close to ideal Proxmox VE and Ceph hardware: identical nodes, SSD and HDD in every sled, dual 10GbE, and IPMI on each one. Ours now runs a good chunk of our internal workload. This is the order we built it in, plus what I'd plan differently next time.Have these lined up firstThe block itself. We used four nodes, and three is the minimum for a sane Ceph cluster.IPMI access to each node for console and virtual mediaThe Proxmox VE ISO, on USB or mounted over IPMIA switch with VLAN support, ideally 10GbE for CephAn IP plan for each network below, and working NTPNetworks first, because moving Ceph later hurtsChanging Ceph networks after the fact is painful, so we settled them before installing anything. We use four logical networks, each on its own VLAN:Management (for example 10.10.10.0/24) carries the Proxmox web UI, SSH and corosync. Corosync is latency-sensitive, so keep it off any link Ceph is saturating. A second corosync link on another network is cheap insurance.Ceph public (10.10.30.0/24) is where monitors and clients talk to OSDs.Ceph cluster (10.10.40.0/24) carries OSD-to-OSD replication and recovery. This is the link that gets hammered when a disk fails.VM traffic rides a VLAN-aware bridge carrying the guest networks.On a two-port 10GbE node, we put Ceph public and cluster on one port, management and VM traffic on the other, and used the 1GbE ports as a corosync backup.Disks: Proxmox on the boot device, everything else to CephNutanix nodes usually boot from a small SATADOM or M.2 device and keep the data drives behind an HBA. Install Proxmox on the boot device and leave every data disk for Ceph. Give each data disk its own OSD and don't put RAID underneath it, since Ceph is your redundancy layer.Keep SSDs and HDDs in separate pools with separate CRUSH rules. If you mix them in one pool, the slowest disk sets the pace for everything. And set expectations with whoever is planning capacity: with the default 3-way replication, usable space is roughly a third of raw.Install every node, then ping everythingInstall from the ISO on each node with a unique hostname and a static management IP. After first boot, set up your repositories, update, and build the bridges and VLAN interfaces in /etc/network/interfaces or the GUI. Before you go further, make sure every node can ping every other node on the management, Ceph public and Ceph cluster networks. A VLAN typo is a lot easier to find now than after Ceph is running on top of it.Building the clusterOn the first node:pvecm create hx-clusterThen join the rest using the first node's management IP:pvecm add 10.10.10.11 pvecm statuspvecm status should list every node and show Quorate: Yes. I join nodes one at a time and let each one settle before adding the next.Ceph packages, monitors and OSDsInstall the Ceph packages on every node, then initialize the config once with both networks:# on every node pveceph install --repository no-subscription # once, on the first node pveceph init --network 10.10.30.0/24 --cluster-network 10.10.40.0/24Monitors need a majority to keep quorum, so run an odd number of them. On four nodes that means three. Run these on three different nodes:pveceph mon create pveceph mgr createThen create one OSD per data disk on each node. Ceph assigns the device class (hdd, ssd or nvme) by itself, and you can check it later with ceph osd tree.lsblk -o NAME,SIZE,ROTA,MODEL pveceph osd create /dev/sdb pveceph osd create /dev/sdc # ...repeat for every data disk on every nodeIf a disk refuses to become an OSDDisks from the previous platform still carry its metadata, and old partitions block OSD creation. Wipe them first with ceph-volume lvm zap /dev/sdX --destroy. Triple-check the device name before you hit enter.Pools split by disk typeCreate one replicated rule per device class, then a pool for each. The --add_storages flag registers the pool as Proxmox storage across the cluster.ceph osd crush rule create-replicated rule-ssd default host ssd ceph osd crush rule create-replicated rule-hdd default host hdd pveceph pool create vm-ssd --crush_rule rule-ssd --add_storages pveceph pool create vm-hdd --crush_rule rule-hdd --add_storagesLeave the defaults of size=3 and min_size=2. We put VM boot disks and databases on the SSD pool and bulk data on the HDD pool.Watch the order here. If you create a pool before your class-specific rules exist, it uses the default rule that spans every disk. You can change it later with ceph osd pool set POOL crush_rule rule-ssd, but that triggers data movement, so it's easier to set the rule when you create the pool.CephFS for the ISO pileRBD pools hold disk images, but ISOs, container templates and backups-in-transit need a filesystem. Put a metadata server on at least two nodes for failover, then create the filesystem:pveceph mds create # on two or more nodes pveceph fs create --name cephfs --add-storageEvery node now sees the same ISO and template library. It's a small thing, but it's one of my favorite quality-of-life wins over local storage.Before you trust it with real VMsceph -s ceph osd tree ceph osd df pvesm statusYou want HEALTH_OK, all monitors in quorum, every OSD up and in, and placement groups active+clean. Then break it on purpose. Migrate a VM between nodes, then reboot one node and watch ceph -s go degraded and recover on its own.A few more things I'd tell the next personUpdate BMC and BIOS firmware while the nodes are still empty, and confirm the HBA presents disks directly to the OS.Point chrony at a reliable NTP source on every node. Monitors complain about clock skew if the node clocks drift.Don't fill it up. Ceph gets unhappy as OSDs approach the nearfull ratio. We plan to stay well under 80 percent so a node failure can re-replicate without hitting full.If I were doing it again, I'd spend even longer on the network plan, since that's the one decision here that's genuinely painful to undo.
Create an account or sign in to comment