Skip to content
View in the app

A better way to browse. Learn more.

Learn more

We moved our whole cloud VPS fleet onto our own Proxmox cluster, and nobody noticed

Featured Replies

Over a few years we'd piled up a fleet of cloud VPS instances: a monitoring box, a ticketing box, a couple of web hosts, a mail relay, and a scattering of one-off servers. Each was cheap on its own, but together they cost us about $334 a month. We consolidated everything onto our own Proxmox cluster in a colocation rack, and the hosting bill dropped to about $112 a month. That's roughly two thirds less, and it keeps heading toward near zero as the last stragglers move. Honestly, the bigger win for us was having one place to back up, monitor and patch. Here's the method that let us do it without the people we support seeing any downtime.

What we had in place first

  • A Proxmox cluster with enough capacity and a backup target
  • Routable public addresses at the new location. We use a small public block.
  • A router or firewall VM in front of the cluster. We use VyOS, and OPNsense or pfSense work too.
  • Control of DNS for every hostname being moved
  • A spreadsheet. Seriously.

Write down what every box really runs

Before moving anything, log into every VPS and write down what it actually runs, not what you remember it running. For each box we captured:

  • Listening ports and the processes behind them (ss -tlnup)
  • Every hostname pointing at it, from DNS and from web server configs
  • Cron jobs and systemd timers for every user
  • Databases, where their data lives, and how big they are
  • Inbound dependencies like webhooks, firewall allowlists at third parties, and API clients that connect by IP
  • Outbound dependencies like SMTP relays and API keys bound to source IPs

The cron jobs and IP allowlists are what get you. A forgotten nightly report, or a vendor that only accepts connections from your old IP, won't show up until a week after cutover when someone asks where their report went.

One container per service

I get the urge to rebuild the old all-in-one boxes as they were. Resist it. We gave each service its own LXC container (or a VM where it needed one) with its own IP on an internal services VLAN, for example 10.10.20.0/24. Now each service gets backed up, restored, upgraded and moved on its own, and the next migration becomes moving one small thing at a time.

You own the firewall now

Your VPS provider was giving you a firewall for free, even if you never thought about it. Now it's your job. We run a VyOS VM with the public uplink on one side and the internal VLANs on the other. The policy is simple. Inbound is dropped by default, established and related return traffic is allowed, and only the specific ports each published service needs are opened. Internal VLANs can reach the internet, but the internet reaches nothing unless a rule says so.

A single reverse proxy for the web stuff

Most of what we moved was HTTP. Instead of giving every web app a public IP, we run one reverse proxy container and forward ports 80 and 443 to it. Ours is Nginx Proxy Manager, though Caddy, Traefik or plain Nginx are just as good. It terminates TLS, holds the certificates, and routes by hostname to the right internal container, so one public IP can serve dozens of sites.

1:1 NAT for the few that need their own IP

Some things don't fit behind a shared HTTP proxy. Think mail servers that need matching reverse DNS, VPN endpoints, game or voice servers on fixed ports, and anything a third party allowlists by IP. For those we use 1:1 NAT on the router, mapping a dedicated public address to one internal container, with firewall rules limited to the ports that service uses. Keep this list short, because every 1:1 mapping is one more exposed surface to watch.

Test the new side while DNS still points at the old one

This is the step that made our cutovers boring, which is exactly what you want. Copy the data, bring the service up on the new side, and test it while DNS still points at the old server. Lower the DNS TTL to 300 seconds a day or two ahead so the eventual flip propagates quickly.

From inside the network, hit the backend directly with a Host header:

curl -sI -H 'Host: app.contoso.com' http://10.10.20.15/

Then test the full public path, including the router, reverse proxy and TLS certificate, without changing DNS. --resolve pins the hostname to the new public IP:

curl -sv --resolve app.contoso.com:443:192.0.2.10 https://app.contoso.com/ -o /dev/null 2>&1 | grep -E 'HTTP/|subject:|expire'

You can do the same thing in a browser by temporarily adding the name to your hosts file. Log in, click around, submit a form, and check that background jobs run. Only once it works end to end do we touch DNS.

Flip DNS, but keep a way back

For services with a database, do a final sync in a quiet window. Stop the app on the old side, take a last dump, restore it on the new side, then flip DNS. After that:

  • Stop the old VPS. Don't delete it. A stopped instance can be started again in a minute if something was missed.
  • Take a provider snapshot of the old box as a second rollback and keep it until you're confident. Snapshots usually cost a fraction of a running instance.
  • If you can, watch the old box's logs for a few days before stopping it. Any traffic still arriving points you at a hard-coded IP or a forgotten client.

Signs the move actually stuck

  • dig +short app.contoso.com returns the new address from several resolvers.
  • The certificate on the new proxy is valid and auto-renewing.
  • Cron jobs, scheduled reports and outbound email are firing from the new location.
  • Each new container shows up in your backup jobs and monitoring.

Things that surprised us

  • Several integrations called our old server by IP rather than hostname. The access logs on the box we kept running a little longer are how we found them.
  • Apps behind the proxy suddenly see every request coming from the proxy's IP, which breaks rate limiting and fail2ban. Configure trusted proxies and X-Forwarded-For handling in each app.
  • A new public IP has no sending history and probably no matching PTR record. Fix reverse DNS and SPF before moving anything that sends mail.
  • Don't cancel the VPS account first. Cancel the instances, keep the snapshots, and close the account only once you no longer need them.

Moving one small service at a time felt slow while we were doing it. Looking back, it's the reason nobody had to call us about it.

Create an account or sign in to comment

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.