Officers had been working around it for a while, and every fix tried so far had been on the laptops themselves, like hosts files, manual DNS settings and re-joining the domain. None of it stuck.

Asking each DNS server directly

From a vehicle laptop we ran the same lookup twice. The first went to the DNS server the laptop had been given by DHCP, and the second went straight to a domain controller's address across the tunnel.

• Querying the domain controller directly worked perfectly. Domain records, SRV records for the DCs, all of it.
• Querying the address handed out by DHCP returned NXDOMAIN for anything in the internal .local zone.

The DHCP-assigned DNS server was the cellular router itself. Its built-in DNS forwarder was answering internal queries on its own instead of passing them up the tunnel. Two lookups, side by side, pointed straight at it.

Why the router said no

The routers had a local domain setting used for naming DHCP clients, and its configured value overlapped the city's internal .local zone. As best we could tell, that made the router's forwarder treat itself as authoritative for those names. Anything not in its own lease table got an immediate NXDOMAIN, and the query never reached the domain controllers. The laptops trusted the answer, and Windows caches negative responses, which made the symptoms even less consistent from one car to the next.

What we changed

1. We changed the routers' local domain setting fleet-wide through the vendor's cloud management platform, to a neutral value that doesn't overlap the AD domain. Every vehicle router was in the same configuration group, so this was one change pushed to the whole fleet instead of a visit to every car.
2. We added AD sites and subnets for the vehicle address ranges. Without them, the domain controllers couldn't map the laptops to a site, so DC location was unpredictable and a laptop could pick a DC that was slow or unreachable over the tunnel.
3. We created reverse lookup zones for the vehicle subnets, so the laptops could register PTR records and tools that rely on reverse lookups behaved properly.

Where it stands

The vehicle laptops resolve internal names reliably over LTE now. Group Policy applies in the field, and domain sign-in works instead of falling back to cached credentials. Domain controllers associate the laptops with the right site, so DC selection is predictable. The fix lives in one place, the router configuration group, so new vehicles get it automatically.

A couple of things I'll carry forward. Small and mobile routers often run a local forwarder with features like local domains and rebind protection that can get in the way of internal zones, especially .local. Any time a client on the same router platform has odd DNS problems, that setting is the first thing I check. And whenever a new address range shows up, mobile and VPN ranges included, it goes into Sites and Services the day it's created, not the day something breaks.

What it runs on

• A Debian 13 LXC container on our Proxmox cluster. It's one service per container, same as everything else we host.
• nginx and PHP 8.4 (PHP-FPM) from the Debian packages.
• MySQL 8.4 running in Docker inside the container, with its socket directory bind-mounted so PHP on the host connects over a local Unix socket rather than TCP.
• Redis for caching and sessions.
• Our existing reverse proxy in front, terminating TLS with a Let's Encrypt certificate.

Putting MySQL in Docker was deliberate. We wanted a specific MySQL version that doesn't depend on the distribution's packaging, and upgrades that are a container image tag instead of a package migration.

A script instead of a checklist

The whole build is one shell script. It creates the container, installs packages, configures nginx and PHP-FPM, starts MySQL with its data on a dedicated volume, creates the database and user, installs Redis, and sets PHP limits for uploads and memory. The only manual part is running Invision's web installer and entering the licence. If the container dies, we can rebuild it and restore the data in well under an hour. Writing the script took longer than doing the install by hand would have, but only the first time.

Making it look like ours

We themed it to the Harbor Tech Bytes logo colours and set up separate light and dark logos, because a logo that looks right on white often disappears on a dark background. The palette is defined once in the theme settings, so headers, buttons and links stay consistent in both modes.

Backups

A nightly job dumps the database with a consistent single-transaction dump, archives the uploads directory and configuration, and ships both off the container. On top of that, Proxmox backs up the whole container every night, so we can restore either a single database or the entire thing. We've tested the restore, because I don't count a backup as real until I've restored from it.

So now we have a forum that's fast, searchable and ours, running in a single container next to the rest of our infrastructure. There's a rebuild script and a tested restore, branding that holds up in light and dark mode, and you're reading the result. If you're running something like this yourself, my main advice is to check the PHP and nginx logs before you start debugging the installer.

On paper the design was fine. Domain laptops with a valid machine certificate get a full network tunnel, and everything else gets remote desktop only. We needed to work out why every laptop was being treated as "everything else".

Following one person's session

We didn't start with overall connection stats. We followed a single user's session from start to finish in the appliance's per-session logs. The access policy ran its steps in order. It authenticated against AD, checked the client's machine certificate, ran an endpoint antivirus check, and then assigned resources. The certificate check failed on every session, on every laptop, every time. The policy then sent the user down the fallback branch that only granted the remote-desktop link, and the client tunnel got torn down because no network-access resource had been assigned. The aggregate numbers just said connections were succeeding. Only the session log showed which step failed and which branch the user ended up on.

The laptops were fine

The obvious theory was that the laptops didn't have certificates. They did. Every domain laptop had a valid computer certificate, auto-enrolled from the client's internal AD Certificate Services root, current, and with client-authentication usage. I'm glad we checked first, because otherwise we could easily have spent days rebuilding certificate templates that were already working.

An empty trust profile

The certificate check on the appliance pointed at a certificate-authority profile, which tells it which CA certificates to trust when validating a client certificate. That profile existed and was attached to the policy, but it had no CA certificate loaded at all, so it could never validate anything. Years of "the VPN is flaky" came down to that one empty field.

What we changed

1. We loaded the internal root CA into the trust profile used by the certificate check.
2. We moved the group check to the front of the policy. While mapping the policy we found a dormant branch. Once the certificate check started passing, a valid machine certificate would have led to a full tunnel without checking VPN group membership on that path. If we'd only fixed the CA, anyone with a domain laptop would have had full access. With the AD group check first, users outside the group never reach the certificate step.
3. We softened the antivirus check. A failed AV check used to deny access entirely, which locked people out when their definitions were a day behind. Now it falls back to remote-desktop-only, so the person can still work and the problem shows up in the logs.
4. We fixed the public certificate chain. The appliance's public TLS certificate had been installed without its intermediate. Most browsers coped by fetching it themselves, but some clients didn't, and that explained a separate batch of intermittent connection errors. We installed the full chain.
5. We documented it all, including the policy flow as a diagram, which CA the check trusts, where the certificates come from, and when each one expires.

Where it stands now

Domain laptops in the VPN group get a full tunnel with mapped drives and internal apps, for the first time since the appliance went in. Users outside the group are denied correctly and consistently, instead of by accident because of a broken check. The access hole that would have opened the moment someone "fixed" the certificate check got closed before it ever went live. The help desk also stopped getting a whole category of tickets.

The part I'd pass on is about fixing a check that has always failed. Before you turn it on, read every branch it feeds. A check that always fails hides whatever is sitting on its success path, and nobody has looked at that path in years.

We also had retired hyperconverged hardware sitting around, a four-node chassis that came out of a customer refresh and was perfectly healthy. So the plan was to move everything onto hardware we own, in a colo rack, while our clients' tickets, alerts and logins kept working the whole time.

The platform

We wiped the four nodes and installed Proxmox VE with Ceph across all of them, so any VM or container can run on any node and survive a node failure. The vendor's own hypervisor stack went away. Proxmox gave us clustering, live migration and built-in backups, with no licensing to keep track of.

How the network is laid out

• A VyOS VM is the router and firewall. It holds a public /28 that the colo routes to us.
• Services live on an internal VLAN behind it. Management interfaces sit on a separate VLAN that isn't reachable from the internet at all.
• Nginx Proxy Manager takes ports 80 and 443 on one public IP and terminates TLS for every HTTP service.
• The few services that really need their own address get a 1:1 NAT from a dedicated public IP on the /28. That's mainly the RMM, which has its own agent ports, and the SIEM, for agent enrolment and event ports.

One service per container

Each service got its own LXC container, or a full VM where the software insisted (mainly the RMM and the SIEM). Monitoring (LibreNMS and Grafana), the wiki, the password manager, the ticketing system and each small app all became separate containers. Snapshots, restores and resource limits are now per-service, and a bad PHP upgrade in one app can't take another one down with it.

Moving things without anyone noticing

We had one rule for every migration. Prove the new origin works before you touch DNS.

1. Build the new container and restore data from the old server.
2. Test it by pointing a hosts-file entry, or a curl with an explicit resolve, at the new address. That way you're exercising the real hostname and certificate.
3. Do a final data sync during a short write freeze where needed (the ticketing and password manager databases).
4. Flip DNS. We'd set low TTLs a day ahead, so most clients moved within minutes.
5. Leave the old server running, read-only where possible, until traffic to it stops.

Nothing was deleted on cutover day. We shut the old servers down and kept provider snapshots for rollback, and only destroyed each one once its service had run cleanly on the new cluster for a while. Those snapshots cost a few dollars, and it was very nice knowing we could roll back.

Where it landed

Monthly hosting went from about $334 to about $112, and most of what's left is leftover services we're still winding down. The goal is close to zero beyond colo and transit. Every service is now backed up, monitored and patched the same way, on the same schedule. Restores are faster too. A container snapshot rollback takes seconds, and backups are local instead of being pulled across the internet. Clients didn't see any outages during the move.

If you're planning something similar, start with the inventory. Our first list of what ran where was missing about a third of the cron jobs, webhooks and little scripts living on those servers. And go in knowing that owning the hardware means owning its problems. We got the cost savings and the control, and we also took on power, spare parts and hardware monitoring. I'd rather plan for a node dying now than figure it out the night it happens.