We moved our whole hosting footprint onto our own Proxmox cluster

Like a lot of small MSPs, our hosting grew one server at a time. We had a VPS for the ticketing system, another for the monitoring stack, and one for the RMM because it wanted its own box. The SIEM got another because it wanted a lot of RAM, a couple more ran WordPress sites and small internal apps, and the documentation wiki and password manager got squeezed in wherever there was room. Each decision made sense at the time. Together they came to about $334 a month across two providers, with backups, patching and firewall rules done slightly differently on every box. Whenever something broke, the first question was always "which server is that one on again?"

We also had retired hyperconverged hardware sitting around, a four-node chassis that came out of a customer refresh and was perfectly healthy. So the plan was to move everything onto hardware we own, in a colo rack, while our clients' tickets, alerts and logins kept working the whole time.

The platform

We wiped the four nodes and installed Proxmox VE with Ceph across all of them, so any VM or container can run on any node and survive a node failure. The vendor's own hypervisor stack went away. Proxmox gave us clustering, live migration and built-in backups, with no licensing to keep track of.

How the network is laid out

• A VyOS VM is the router and firewall. It holds a public /28 that the colo routes to us.
• Services live on an internal VLAN behind it. Management interfaces sit on a separate VLAN that isn't reachable from the internet at all.
• Nginx Proxy Manager takes ports 80 and 443 on one public IP and terminates TLS for every HTTP service.
• The few services that really need their own address get a 1:1 NAT from a dedicated public IP on the /28. That's mainly the RMM, which has its own agent ports, and the SIEM, for agent enrolment and event ports.

One service per container

Each service got its own LXC container, or a full VM where the software insisted (mainly the RMM and the SIEM). Monitoring (LibreNMS and Grafana), the wiki, the password manager, the ticketing system and each small app all became separate containers. Snapshots, restores and resource limits are now per-service, and a bad PHP upgrade in one app can't take another one down with it.

Moving things without anyone noticing

We had one rule for every migration. Prove the new origin works before you touch DNS.

1. Build the new container and restore data from the old server.
2. Test it by pointing a hosts-file entry, or a curl with an explicit resolve, at the new address. That way you're exercising the real hostname and certificate.
3. Do a final data sync during a short write freeze where needed (the ticketing and password manager databases).
4. Flip DNS. We'd set low TTLs a day ahead, so most clients moved within minutes.
5. Leave the old server running, read-only where possible, until traffic to it stops.

Nothing was deleted on cutover day. We shut the old servers down and kept provider snapshots for rollback, and only destroyed each one once its service had run cleanly on the new cluster for a while. Those snapshots cost a few dollars, and it was very nice knowing we could roll back.

Where it landed

Monthly hosting went from about $334 to about $112, and most of what's left is leftover services we're still winding down. The goal is close to zero beyond colo and transit. Every service is now backed up, monitored and patched the same way, on the same schedule. Restores are faster too. A container snapshot rollback takes seconds, and backups are local instead of being pulled across the internet. Clients didn't see any outages during the move.

If you're planning something similar, start with the inventory. Our first list of what ran where was missing about a third of the cron jobs, webhooks and little scripts living on those servers. And go in knowing that owning the hardware means owning its problems. We got the cost savings and the control, and we also took on power, spare parts and hardware monitoring. I'd rather plan for a node dying now than figure it out the night it happens.