Keaton

Hey, and welcome to TechBytes. This started as our own notes: the fixes, the configs and the "never doing that again" moments from supporting cities, small businesses and our own servers. We figured they'd be more useful out in the open. A good place to start is Tutorials or Show & Tell. If you've got a minute, say hi in Introductions and tell us what you work on. And if you're stuck on something, post in Help & Q&A. That's what it's there for. Glad you're here. Keaton

One of the small cities we support has a rugged laptop in each police vehicle, connected through an in-car cellular router. The router brings up a tunnel back to the city network so officers can reach dispatch, records systems and file shares from the car. The tunnel was up, and pinging internal servers by IP worked. Even so, the laptops couldn't resolve anything in the city's internal Active Directory domain. Group Policy failed, domain authentication fell back to cached credentials, and any application that referred to a server by name broke. Officers had been working around it for a while, and every fix tried so far had been on the laptops themselves, like hosts files, manual DNS settings and re-joining the domain. None of it stuck. Asking each DNS server directlyFrom a vehicle laptop we ran the same lookup twice. The first went to the DNS server the laptop had been given by DHCP, and the second went straight to a domain controller's address across the tunnel. Querying the domain controller directly worked perfectly. Domain records, SRV records for the DCs, all of it.Querying the address handed out by DHCP returned NXDOMAIN for anything in the internal .local zone.The DHCP-assigned DNS server was the cellular router itself. Its built-in DNS forwarder was answering internal queries on its own instead of passing them up the tunnel. Two lookups, side by side, pointed straight at it. Why the router said noThe routers had a local domain setting used for naming DHCP clients, and its configured value overlapped the city's internal .local zone. As best we could tell, that made the router's forwarder treat itself as authoritative for those names. Anything not in its own lease table got an immediate NXDOMAIN, and the query never reached the domain controllers. The laptops trusted the answer, and Windows caches negative responses, which made the symptoms even less consistent from one car to the next. What we changedWe changed the routers' local domain setting fleet-wide through the vendor's cloud management platform, to a neutral value that doesn't overlap the AD domain. Every vehicle router was in the same configuration group, so this was one change pushed to the whole fleet instead of a visit to every car.We added AD sites and subnets for the vehicle address ranges. Without them, the domain controllers couldn't map the laptops to a site, so DC location was unpredictable and a laptop could pick a DC that was slow or unreachable over the tunnel.We created reverse lookup zones for the vehicle subnets, so the laptops could register PTR records and tools that rely on reverse lookups behaved properly.Rollout notes Negative caching made it look like the fix hadn't worked. After the router change, some laptops still failed because they'd cached the NXDOMAIN answers. Flushing the DNS client cache, or just rebooting, cleared it. We put that in the rollout notes so nobody would decide the fix was a dud.A few vehicles were parked with the router powered off, so not every router picked up the change at once. They took the new configuration when they next came online. We tracked the stragglers from the management portal instead of assuming we'd hit 100%.We almost fixed the wrong layer. Before the side-by-side test, the plan was to push static DNS servers to the laptops through Group Policy. That would have worked on the tunnel and failed badly as soon as a laptop was on any other network. Fixing the router kept DHCP as the single source of truth.Where it standsThe vehicle laptops resolve internal names reliably over LTE now. Group Policy applies in the field, and domain sign-in works instead of falling back to cached credentials. Domain controllers associate the laptops with the right site, so DC selection is predictable. The fix lives in one place, the router configuration group, so new vehicles get it automatically. A couple of things I'll carry forward. Small and mobile routers often run a local forwarder with features like local domains and rebind protection that can get in the way of internal zones, especially .local. Any time a client on the same router platform has odd DNS problems, that setting is the first thing I check. And whenever a new address range shows up, mobile and VPN ranges included, it goes into Sites and Services the day it's created, not the day something breaks.

We wanted somewhere to share what we learn on the job. That means how-tos, project write-ups, and the gotchas that cost us an afternoon, so hopefully they don't cost you one. A static blog felt too one-way, and a chat server buries everything within a week. We wanted a proper forum with search, tags and accounts, which led us to Invision Community. It had to run on our own infrastructure, be easy to rebuild, look like ours, and be backed up properly. What it runs onA Debian 13 LXC container on our Proxmox cluster. It's one service per container, same as everything else we host.nginx and PHP 8.4 (PHP-FPM) from the Debian packages.MySQL 8.4 running in Docker inside the container, with its socket directory bind-mounted so PHP on the host connects over a local Unix socket rather than TCP.Redis for caching and sessions.Our existing reverse proxy in front, terminating TLS with a Let's Encrypt certificate.Putting MySQL in Docker was deliberate. We wanted a specific MySQL version that doesn't depend on the distribution's packaging, and upgrades that are a container image tag instead of a package migration. A script instead of a checklistThe whole build is one shell script. It creates the container, installs packages, configures nginx and PHP-FPM, starts MySQL with its data on a dedicated volume, creates the database and user, installs Redis, and sets PHP limits for uploads and memory. The only manual part is running Invision's web installer and entering the licence. If the container dies, we can rebuild it and restore the data in well under an hour. Writing the script took longer than doing the install by hand would have, but only the first time. Making it look like oursWe themed it to the Harbor Tech Bytes logo colours and set up separate light and dark logos, because a logo that looks right on white often disappears on a dark background. The palette is defined once in the theme settings, so headers, buttons and links stay consistent in both modes. BackupsA nightly job dumps the database with a consistent single-transaction dump, archives the uploads directory and configuration, and ships both off the container. On top of that, Proxmox backs up the whole container every night, so we can restore either a single database or the entire thing. We've tested the restore, because I don't count a backup as real until I've restored from it. Where the install fought back An installer error that wasn't about the installerThe web installer failed at the database step with an error that looked like a problem with the installer itself. It turned out PHP couldn't find the MySQL socket. PHP was looking at the default socket path, and with MySQL in a container the socket only shows up on the host if the directory is bind-mounted and the permissions let PHP-FPM's user reach it. Once we shared the socket directory properly and set the path explicitly in the configuration, the database step passed. Then the installer stalled at a session check, where it verifies that PHP sessions persist between requests. That one was on the PHP side too. Sessions weren't being saved anywhere the next request could read them. Once PHP had a working, writable session store, re-running the installer got us through. Both failures were obvious in the PHP-FPM and nginx error logs and misleading on screen. An installer tells you which step failed, not why, so check the logs before you believe the message. A custom page that threw a server errorWe built a custom page for the site and it returned a server error instead of content. Friendly (clean) URLs weren't enabled in our install, so nothing could resolve the page's pretty URL. Turning on friendly URLs in the admin panel, along with the matching nginx rewrite that routes requests to Invision's front controller, fixed it. The rest of the site's URLs got nicer as a bonus. Behind the proxyLike every app behind a reverse proxy, the forum first saw every visitor as coming from the proxy's address. Configuring nginx to trust the proxy's forwarded headers gave us real client IPs back, which matters for spam protection and moderation. So now we have a forum that's fast, searchable and ours, running in a single container next to the rest of our infrastructure. There's a rebuild script and a tested restore, branding that holds up in light and dark mode, and you're reading the result. If you're running something like this yourself, my main advice is to check the PHP and nginx logs before you start debugging the installer.

One of our municipal clients had a remote-access VPN appliance that, as far as anyone could remember, had never worked properly. Staff in the VPN group would connect and get a portal page with a single remote-desktop link. If they launched the full VPN client instead, it connected and then dropped within about a second. Over the years people had just learned to use the remote-desktop link and live without mapped drives, internal web apps and anything else that needed a real tunnel. When we asked about it, the answer was basically "oh, the VPN's always been like that." On paper the design was fine. Domain laptops with a valid machine certificate get a full network tunnel, and everything else gets remote desktop only. We needed to work out why every laptop was being treated as "everything else". Following one person's sessionWe didn't start with overall connection stats. We followed a single user's session from start to finish in the appliance's per-session logs. The access policy ran its steps in order. It authenticated against AD, checked the client's machine certificate, ran an endpoint antivirus check, and then assigned resources. The certificate check failed on every session, on every laptop, every time. The policy then sent the user down the fallback branch that only granted the remote-desktop link, and the client tunnel got torn down because no network-access resource had been assigned. The aggregate numbers just said connections were succeeding. Only the session log showed which step failed and which branch the user ended up on. The laptops were fineThe obvious theory was that the laptops didn't have certificates. They did. Every domain laptop had a valid computer certificate, auto-enrolled from the client's internal AD Certificate Services root, current, and with client-authentication usage. I'm glad we checked first, because otherwise we could easily have spent days rebuilding certificate templates that were already working. An empty trust profileThe certificate check on the appliance pointed at a certificate-authority profile, which tells it which CA certificates to trust when validating a client certificate. That profile existed and was attached to the policy, but it had no CA certificate loaded at all, so it could never validate anything. Years of "the VPN is flaky" came down to that one empty field. What we changedWe loaded the internal root CA into the trust profile used by the certificate check.We moved the group check to the front of the policy. While mapping the policy we found a dormant branch. Once the certificate check started passing, a valid machine certificate would have led to a full tunnel without checking VPN group membership on that path. If we'd only fixed the CA, anyone with a domain laptop would have had full access. With the AD group check first, users outside the group never reach the certificate step.We softened the antivirus check. A failed AV check used to deny access entirely, which locked people out when their definitions were a day behind. Now it falls back to remote-desktop-only, so the person can still work and the problem shows up in the logs.We fixed the public certificate chain. The appliance's public TLS certificate had been installed without its intermediate. Most browsers coped by fetching it themselves, but some clients didn't, and that explained a separate batch of intermittent connection errors. We installed the full chain.We documented it all, including the policy flow as a diagram, which CA the check trusts, where the certificates come from, and when each one expires.Things that slowed us down On many appliances of this type, saving a policy edit doesn't make it live until you explicitly apply the policy. We now apply, then confirm the change actually took effect in the next session log instead of trusting the editor.Users with existing sessions kept the old behaviour until their session was killed. We cleared the test users' sessions between every test.The fix depends on auto-enrolment staying healthy. A laptop that sits off the network long enough for its certificate to expire drops back to remote-desktop-only. That's the correct behaviour, but it's something to watch now, so we added certificate expiry on long-offline devices to our regular checks.Where it stands nowDomain laptops in the VPN group get a full tunnel with mapped drives and internal apps, for the first time since the appliance went in. Users outside the group are denied correctly and consistently, instead of by accident because of a broken check. The access hole that would have opened the moment someone "fixed" the certificate check got closed before it ever went live. The help desk also stopped getting a whole category of tickets. The part I'd pass on is about fixing a check that has always failed. Before you turn it on, read every branch it feeds. A check that always fails hides whatever is sitting on its success path, and nobody has looked at that path in years.

Like a lot of small MSPs, our hosting grew one server at a time. We had a VPS for the ticketing system, another for the monitoring stack, and one for the RMM because it wanted its own box. The SIEM got another because it wanted a lot of RAM, a couple more ran WordPress sites and small internal apps, and the documentation wiki and password manager got squeezed in wherever there was room. Each decision made sense at the time. Together they came to about $334 a month across two providers, with backups, patching and firewall rules done slightly differently on every box. Whenever something broke, the first question was always "which server is that one on again?" We also had retired hyperconverged hardware sitting around, a four-node chassis that came out of a customer refresh and was perfectly healthy. So the plan was to move everything onto hardware we own, in a colo rack, while our clients' tickets, alerts and logins kept working the whole time. The platformWe wiped the four nodes and installed Proxmox VE with Ceph across all of them, so any VM or container can run on any node and survive a node failure. The vendor's own hypervisor stack went away. Proxmox gave us clustering, live migration and built-in backups, with no licensing to keep track of. How the network is laid outA VyOS VM is the router and firewall. It holds a public /28 that the colo routes to us.Services live on an internal VLAN behind it. Management interfaces sit on a separate VLAN that isn't reachable from the internet at all.Nginx Proxy Manager takes ports 80 and 443 on one public IP and terminates TLS for every HTTP service.The few services that really need their own address get a 1:1 NAT from a dedicated public IP on the /28. That's mainly the RMM, which has its own agent ports, and the SIEM, for agent enrolment and event ports.One service per containerEach service got its own LXC container, or a full VM where the software insisted (mainly the RMM and the SIEM). Monitoring (LibreNMS and Grafana), the wiki, the password manager, the ticketing system and each small app all became separate containers. Snapshots, restores and resource limits are now per-service, and a bad PHP upgrade in one app can't take another one down with it. Moving things without anyone noticingWe had one rule for every migration. Prove the new origin works before you touch DNS. Build the new container and restore data from the old server.Test it by pointing a hosts-file entry, or a curl with an explicit resolve, at the new address. That way you're exercising the real hostname and certificate.Do a final data sync during a short write freeze where needed (the ticketing and password manager databases).Flip DNS. We'd set low TTLs a day ahead, so most clients moved within minutes.Leave the old server running, read-only where possible, until traffic to it stops.Nothing was deleted on cutover day. We shut the old servers down and kept provider snapshots for rollback, and only destroyed each one once its service had run cleanly on the new cluster for a while. Those snapshots cost a few dollars, and it was very nice knowing we could roll back. What tripped us up Apps that didn't trust the proxy. Several applications, LibreNMS included, generated wrong redirect URLs or treated every visitor as coming from the proxy's IP until we told them which proxies to trust. Each app has its own setting for this, and checking for it is now part of every migration.Old addresses were hard-coded all over the place. Webhooks, alert transports and integrations still pointed at the old servers' IPs or hostnames. Monitoring alerts went quiet for a while because a webhook target was still the old box, which is not a great feeling for an MSP. Now we grep every config and database for old IPs and hostnames before decommissioning anything.The RMM expected a certificate covering several subdomains, and a per-host certificate caused agent connection errors. A wildcard certificate fixed it.Adding an interface to the VyOS VM changed the order the guest saw its NICs, which swapped our inside and outside networks on reboot. We pin interfaces to MAC addresses in the router config now.Moving lots of data over SSH tripped fail2ban on the old servers, so we were locking ourselves out. Batching commands into fewer connections and whitelisting our office IP sorted it.Where it landedMonthly hosting went from about $334 to about $112, and most of what's left is leftover services we're still winding down. The goal is close to zero beyond colo and transit. Every service is now backed up, monitored and patched the same way, on the same schedule. Restores are faster too. A container snapshot rollback takes seconds, and backups are local instead of being pulled across the internet. Clients didn't see any outages during the move. If you're planning something similar, start with the inventory. Our first list of what ran where was missing about a third of the cron jobs, webhooks and little scripts living on those servers. And go in knowing that owning the hardware means owning its problems. We got the cost savings and the control, and we also took on power, spare parts and hardware monitoring. I'd rather plan for a node dying now than figure it out the night it happens.

Bulk renames look trivial. Loop over the files, work out a new name, move. The trouble starts when two inputs map to the same output, because a plain mv or Move-Item -Force overwrites the first file with the second and says nothing. We learned this the hard way normalising a folder of scanned documents for a client. Invoice 001.pdf and invoice_001.pdf both became invoice_001.pdf, and one of them was simply gone. These days every batch job I write follows the same habits. Here they are, with working bash and PowerShell versions. Before you run either script bash 4 or newer for the associative array in the bash version. macOS ships bash 3.2, so install a newer bash or drop the collision map.PowerShell 5.1 or 7.x for the PowerShell version.A copy of the data, or at least a backup, for the first real run.The rules I follow nowDry run by default. The script only prints what it would do unless you explicitly tell it to act.Detect collisions in the plan, before anything moves.Never clobber. Check the destination, and use mv -n as a backstop.Log every change with old and new names, so you can undo it.Make it re-runnable. Files already in the target state get skipped, so running it twice is harmless and an interrupted run can just be restarted.Bash#!/usr/bin/env bash # normalize-names.sh - lowercase names and replace spaces with underscores # Usage: ./normalize-names.sh DIR (dry run) # ./normalize-names.sh DIR --apply (actually rename) set -euo pipefail DIR="${1:?usage: $0 DIR [--apply]}" APPLY=0 [[ "${2:-}" == "--apply" ]] && APPLY=1 LOG="rename-$(date +%Y%m%d-%H%M%S).log" declare -A planned # target -> source, needs bash 4+ while IFS= read -r -d '' src; do dir=$(dirname "$src"); base=$(basename "$src") new=$(printf '%s' "$base" | tr '[:upper:]' '[:lower:]' | tr ' ' '_') [[ "$base" == "$new" ]] && continue # already done: re-runnable dest="$dir/$new" if [[ -n "${planned[$dest]:-}" ]]; then echo "COLLISION $src and ${planned[$dest]} -> $dest" | tee -a "$LOG" continue fi planned[$dest]="$src" if [[ -e "$dest" ]]; then echo "EXISTS $src -> $dest (skipped)" | tee -a "$LOG" continue fi if (( APPLY )); then mv -n -- "$src" "$dest" if [[ -e "$src" ]]; then echo "FAILED $src -> $dest" | tee -a "$LOG" else echo "RENAMED $src -> $dest" | tee -a "$LOG" fi else echo "WOULD $src -> $dest" | tee -a "$LOG" fi done < <(find "$DIR" -maxdepth 1 -type f -print0)A few details in there are easy to miss: find -print0 with read -d '' handles spaces, quotes and even newlines in names. Never loop over $(ls).mv -n (no clobber) refuses to overwrite an existing file. I don't trust its exit status to tell me it skipped, because that behaviour differs between platforms and coreutils versions, so the script checks whether the source still exists afterwards.The -- before the filenames stops a file called -rf.txt being read as options.PowerShellPowerShell gives you dry runs for free. Declare SupportsShouldProcess, wrap the change in $PSCmdlet.ShouldProcess(), and -WhatIf works just like it does on built-in cmdlets. # Normalize-Names.ps1 - run with -WhatIf first [CmdletBinding(SupportsShouldProcess)] param([Parameter(Mandatory)][string]$Path) $log = "rename-{0:yyyyMMdd-HHmmss}.csv" -f (Get-Date) $plan = Get-ChildItem -LiteralPath $Path -File | ForEach-Object { $new = $_.Name.ToLower() -replace '\s+', '_' [pscustomobject]@{ Source = $_.FullName; NewName = $new; Target = Join-Path $_.DirectoryName $new } } | Where-Object { (Split-Path $_.Source -Leaf) -cne $_.NewName } # skip already-normalized # Two inputs mapping to the same output? Stop before touching anything. $collisions = $plan | Group-Object Target | Where-Object Count -gt 1 if ($collisions) { foreach ($c in $collisions) { Write-Warning "Collision -> $($c.Name): $($c.Group.Source -join ', ')" } throw 'Resolve collisions before renaming.' } $results = foreach ($item in $plan) { if (Test-Path -LiteralPath $item.Target) { [pscustomobject]@{ Source = $item.Source; Target = $item.Target; Result = 'SkippedExists' } continue } if ($PSCmdlet.ShouldProcess($item.Source, "Rename to $($item.NewName)")) { try { Rename-Item -LiteralPath $item.Source -NewName $item.NewName -ErrorAction Stop $r = 'Renamed' } catch { $r = "Failed: $($_.Exception.Message)" } [pscustomobject]@{ Source = $item.Source; Target = $item.Target; Result = $r } } } if ($results) { $results | Export-Csv -LiteralPath $log -NoTypeInformation -WhatIf:$false }Run it in two passes: .\Normalize-Names.ps1 -Path D:\Scans\Incoming -WhatIf .\Normalize-Names.ps1 -Path D:\Scans\IncomingRename-Item already refuses to overwrite an existing file, but we still test first so the skip lands in the log instead of showing up as an error. Keep Move-Item -Force out of rename scripts. -Force is exactly the overwrite you're trying to prevent. Read the plan before you apply itWhen the dry run finishes, I look for any COLLISION or warning lines first. Then any EXISTS lines, since a target that's already there usually means an earlier partial run or a genuine duplicate. Last, I check the count against what I expected. If the plan says it'll rename 4,000 files and you expected 40, stop. Did anything go missing? Count files before and after. The numbers must match. Renames never reduce the file count unless something got overwritten.Run the script again in dry-run mode. It should report nothing to do.Keep the log with the data. To undo, swap the columns and feed it back through the same no-clobber logic.Where this has caught us out Case-only renames on case-insensitive filesystems. On NTFS and default macOS APFS, Report.PDF to report.pdf looks like the destination "already exists" because it's the same file, so both scripts above will skip it. If case-only changes matter, go through a temporary name (Report.PDF to Report.PDF.tmp to report.pdf).The in-script map only catches collisions within one run. The destination-exists check is what covers files left by earlier runs.Write logs somewhere other than the target folder, or the next run may try to rename its own log.OneDrive, Dropbox and DFS replication can lag or create conflict copies during a big rename. Pause sync or run it on the server side.I use the same pattern for other batch jobs too, like bulk AD changes, mailbox moves and clearing out old backups. Show the plan, check for collisions, then act, log it, and make sure a second run won't hurt.

Every AD we inherit has the same ghosts. There are computer accounts for laptops recycled years ago, users who left before the last audit, and "temp" accounts that quietly became permanent. A stale-object report is one of the first things we run in a new environment, and after that we schedule it monthly. Here's the script, plus the caveats that keep you from disabling the PC the front desk still logs into every morning. What you need A domain-joined machine with the RSAT Active Directory module (Import-Module ActiveDirectory).A normal domain user account. It can read these attributes, so you don't need Domain Admin just to report.A share or folder to save the reports to.Why lastLogonTimestamp lies a littleAD has two attributes that matter here. lastLogon is precise but not replicated. Each domain controller keeps its own value, so you'd have to query every DC and take the newest.lastLogonTimestamp is replicated, so one query gives a usable answer, but it's deliberately imprecise. It only updates when the existing value is older than ms-DS-Logon-Time-Sync-Interval (14 days by default) minus a random 0 to 5 days. In practice it can lag the real last logon by roughly 9 to 14 days.For a "not seen in 90 days" report that doesn't matter at all. For "did this user log in yesterday?" it's useless. Pick thresholds well above 14 days. Turning the number into a dateThe attribute is a Windows FILETIME (100-nanosecond intervals since 1 January 1601), and [DateTime]::FromFileTime() converts it to local time. Accounts that have never logged on have no value, and converting 0 gives you a date in 1601, so handle that explicitly. The AD module also exposes a ready-converted LastLogonDate property, which is fine to use. I convert manually so the behaviour is obvious to whoever reads the script after me. The script# Get-StaleADReport.ps1 Import-Module ActiveDirectory $Days = 90 $Cutoff = (Get-Date).AddDays(-$Days) $Stamp = Get-Date -Format 'yyyy-MM-dd' $OutDir = '\\fileserver.contoso.com\IT\Reports\AD' function Convert-LLT($value) { if ($value -and $value -gt 0) { [DateTime]::FromFileTime($value) } else { $null } } $computers = Get-ADComputer -Filter * -Properties LastLogonTimestamp, OperatingSystem, Enabled, Description | ForEach-Object { [pscustomobject]@{ Name = $_.Name Enabled = $_.Enabled OS = $_.OperatingSystem LastLogon = Convert-LLT $_.LastLogonTimestamp Description = $_.Description DN = $_.DistinguishedName } } | Where-Object { -not $_.LastLogon -or $_.LastLogon -lt $Cutoff } | Sort-Object LastLogon $users = Get-ADUser -Filter 'Enabled -eq $true' -Properties LastLogonTimestamp, whenCreated | ForEach-Object { [pscustomobject]@{ SamAccountName = $_.SamAccountName Name = $_.Name LastLogon = Convert-LLT $_.LastLogonTimestamp Created = $_.whenCreated DN = $_.DistinguishedName } } | Where-Object { (-not $_.LastLogon -and $_.Created -lt $Cutoff) -or ($_.LastLogon -and $_.LastLogon -lt $Cutoff) } | Sort-Object LastLogon $computers | Export-Csv "$OutDir\stale-computers-$Stamp.csv" -NoTypeInformation $users | Export-Csv "$OutDir\stale-users-$Stamp.csv" -NoTypeInformation "Stale (> $Days days): $($computers.Count) computers, $($users.Count) enabled users"Look at the user filter. An account that has never logged on but was created yesterday isn't stale, so for those we compare whenCreated instead. Getting the report to someoneWe save dated CSVs to a restricted share so we can compare month to month. If you want it emailed, know that Microsoft has marked Send-MailMessage obsolete and it doesn't handle modern authentication well. For Microsoft 365 tenants we send through Graph's sendMail using an app registration with Mail.Send, scoped with an application access policy so it can only send as the reporting mailbox. A relay you control works too. Either way, attach the CSV or link to the share. Nobody wants hundreds of rows pasted into an email body. Sanity checks before anyone disables anything Pick three machines from the report and look them up in your RMM or endpoint manager. If they've been online recently, something's off. Check whether they're really using a different computer account name.Pick one machine you know is used every day and confirm it's not on the report.Cross-check a handful of users against HR's leaver list.Then go in stages. Disable the object, move it to a "Pending Delete" OU with a note in the description, wait 30 days, and only then delete. Disabling is reversible. Deleting without the Recycle Bin is not. A matching bug that made a report look cleanThis one isn't about lastLogonTimestamp, but it came out of the same kind of cleanup. We were comparing a device export against a client list to find machines assigned to the wrong client. Using -like "*$client*" meant a client called "Riverton" also matched "North Riverton", so devices were silently counted under the wrong client and the report looked fine. Match names exactly, with -eq or a lookup set: # BAD: substring / wildcard match - "Riverton" also matches "North Riverton" $devices | Where-Object { $_.ClientName -like "*$client*" } $devices | Where-Object { $_.ClientName -match $client } # regex, same problem # GOOD: exact, case-insensitive comparison $devices | Where-Object { $_.ClientName -eq $client } # GOOD for large lists: a HashSet with a case-insensitive comparer $known = [System.Collections.Generic.HashSet[string]]::new( [string[]]$clientList.Name, [System.StringComparer]::OrdinalIgnoreCase) $unmatched = $devices | Where-Object { -not $known.Contains($_.ClientName.Trim()) }Accounts that look stale but aren't Service accounts and appliances often authenticate in ways that don't update lastLogonTimestamp the way you'd expect, or they log on rarely. Exclude a known service-account OU or review those by hand.Cluster and virtual objects (cluster name objects, some SQL listener accounts) can look stale while being critical. Read the description and DN before you disable one.Computer password age is a useful second opinion. Domain members change their machine password every 30 days by default, so a PasswordLastSet more than a few months old on a computer account is a strong sign the machine is gone.Hybrid-joined devices may also show up in Entra ID. Clean up both, or the stale device keeps reappearing in your Intune reports.

Running a script on one machine is easy. Running it on four hundred machines across a dozen clients, where half of them are laptops that drift in and out of the office, is how you find out whether the script was any good. We use Tactical RMM (TRMM) for this, but most of what follows carries over to any RMM. It's mostly about knowing what account the script runs as, getting output you can actually read afterwards, and not pushing to everyone on the first try. What to have ready A working Tactical RMM instance with agents deployed.An account that can manage scripts and run them on agents.An API key (Settings > Global Settings > API Keys) if you want to drive it from your own tooling.One or two test machines you're allowed to break.Keep a library, not a pile of one-offsAnything you run more than once goes in the Script Manager with a clear name, a category, a description of what it changes, and a sensible default timeout. TRMM supports PowerShell, batch and Python. Ad-hoc commands from the console are fine for poking around, but if it changes state it belongs in the library, where someone else can review it and re-run it exactly the same way. Remember it runs as SYSTEMBy default TRMM scripts run as SYSTEM. That's what you want for installing software, changing services, editing HKLM or anything else machine-wide. It also means a few things catch people out: HKCU is SYSTEM's hive, not the user's.Mapped drives and user-profile paths aren't what you expect.Network access goes out as the computer account.The Run As User option runs the script in the logged-on user's session instead. It only works when somebody is actually signed in, so it's a poor fit for an overnight fleet run. For per-user settings we usually run as SYSTEM and loop through the user hives, or deploy the change somewhere that applies at sign-in. One script with parameters beats five copiesInstead of keeping five near-identical scripts, accept parameters and pass them from the script's default arguments or at run time: param( [string]$ServiceName = 'Spooler', [ValidateSet('Check','Restart')][string]$Mode = 'Check' )Arguments are typed the way you would on a command line (for example -ServiceName Spooler -Mode Restart). Never pass secrets as plain arguments, because they end up in run history. Use the RMM's custom fields or environment variables where you can, and limit who can view history. Write it so running it twice is harmlessA fleet script will hit machines in every state you can imagine, and someone will run it twice. Check before you change anything, make a second run a no-op, and report clearly. We start every output line with a fixed status word so results can be grouped later, and we always set an exit code: # Ensure-BitLockerRecoveryEscrow.ps1 - safe to run repeatedly $ErrorActionPreference = 'Stop' try { $vol = Get-BitLockerVolume -MountPoint $env:SystemDrive if ($vol.ProtectionStatus -ne 'On') { Write-Output "SKIP: BitLocker not enabled on $env:SystemDrive" exit 0 } $kp = $vol.KeyProtector | Where-Object KeyProtectorType -eq 'RecoveryPassword' if (-not $kp) { Write-Output "FAIL: no recovery password protector" exit 1 } BackupToAAD-BitLockerKeyProtector -MountPoint $env:SystemDrive -KeyProtectorId $kp[0].KeyProtectorId | Out-Null Write-Output "OK: escrowed $($kp[0].KeyProtectorId)" exit 0 } catch { Write-Output "FAIL: $($_.Exception.Message)" exit 1 }Don't skip the exit code. When a script is used as a check or an automated task, the RMM uses the return code to decide pass or fail. A script that catches every error and exits 0 will show green forever, and whoever's watching the dashboard has no idea machines are failing underneath it. Roll it out in ringsOrder matters here. This is the sequence we stick to: Run on one test machine and read the full output.Run on a small, mixed group: a desktop, a laptop, an older OS build, a machine with a non-admin user.Use bulk script against a single client or site.Only then target everything, or attach it to an automation policy.Put recurring work on a policyFor anything recurring, add the script as an automated task on an automation policy instead of running it by hand every week. Policies apply at the client, site or agent level, so you can pilot on one site before widening it. Offline agents pick up their tasks when they come back, which beats chasing laptops around manually. Counting the results Export the bulk results and count statuses. Eyeballing hundreds of rows is how things get missed. # results.csv exported from the bulk run: hostname, output Import-Csv .\results.csv | ForEach-Object { $status = ($_.output -split ':')[0].Trim() [pscustomobject]@{ Host = $_.hostname; Status = $status; Detail = $_.output } } | Group-Object Status | Select-Object Name, CountEvery machine should land in OK, SKIP or FAIL. Anything with no output at all usually timed out or was offline, so rerun those separately. Calling TRMM from your own scriptsOnce you're comfortable, you can trigger runs from your own tooling. Authentication is an X-API-KEY header. Check field names against your version, since they've changed over time. The easiest way I've found is to watch the request the web UI sends in your browser's developer tools. $base = 'https://api.rmm.contoso.com' $headers = @{ 'X-API-KEY' = (Get-Secret -Name 'TrmmApiKey' -AsPlainText) } # Cloudflare Browser Integrity Check blocked our POSTs without this $ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36' # GET worked even without the User-Agent $agents = Invoke-RestMethod -Uri "$base/agents/" -Headers $headers -UserAgent $ua # POST returned 403 / error 1010 until -UserAgent was added $body = @{ script = 42 # script ID from the library args = @('-Mode', 'Check') output = 'wait' timeout = 120 run_as_user = $false } | ConvertTo-Json $agentId = '<AgentID>' Invoke-RestMethod -Method Post -Uri "$base/agents/$agentId/runscript/" ` -Headers $headers -UserAgent $ua -ContentType 'application/json' -Body $bodyThe 1010 error, and a few others Our RMM API sits behind Cloudflare. GET requests worked perfectly, which made the next part confusing. Every POST and DELETE came back 403 with Cloudflare error 1010. Error 1010 means Cloudflare blocked the request based on the client's browser signature, usually via Browser Integrity Check. Scripted clients send a non-browser User-Agent, and only the write requests were being judged that harshly. Sending a normal browser User-Agent header fixed it straight away. The cleaner long-term fix is a WAF rule that skips Browser Integrity Check for the API hostname, restricted to your own source IPs. Timeouts are often too short. Installers on slow laptops easily run past a 30-second default, and a timed-out script may still be running on the endpoint."It works when I run it" usually means it works as you, in your session, with your drives mapped. SYSTEM is a different world.Keep output short and structured. A script that dumps a full event log for every machine makes the results unreadable.

Most of our tenant reporting runs overnight: device inventories, licence counts, stale-account checks. Nobody is sitting there to answer an MFA prompt at 2 a.m., and a job tied to one tech's sign-in breaks the week that person gets a new phone or moves on. So we run these jobs as app registrations using the OAuth2 client credentials flow. The script signs in as the app itself, gets a token, and calls Graph with plain Invoke-RestMethod. No modules are needed, which I appreciate on locked-down jump boxes where installing anything is its own ticket. This is the small script we start from. It lists users and Intune managed devices, follows paging properly, and keeps the secret out of the file. Before you start An account that can create app registrations and grant admin consent (Application Administrator, plus a Privileged Role Administrator or Global Administrator to consent to Graph application permissions).PowerShell 5.1 or 7.x.The Microsoft.PowerShell.SecretManagement and Microsoft.PowerShell.SecretStore modules if you go with a client secret.Registering the appIn the Entra admin center, go to Applications > App registrations > New registration.Give it a name the next tech will understand (we use something like svc-graph-reporting). Single tenant, no redirect URI.Copy the Application (client) ID and Directory (tenant) ID from the Overview page.Permissions, and the consent button everyone forgetsUnder API permissions > Add a permission > Microsoft Graph, pick Application permissions, not Delegated. Delegated permissions are for a signed-in user and do nothing in the client credentials flow. This script needs two: User.Read.AllDeviceManagementManagedDevices.Read.AllThen click Grant admin consent for <tenant>. This is the step I always forget on a new tenant. Until the status column shows a green tick, you'll still get a token, but every call comes back 403. Ask only for the read permissions you actually need. An app secret with Directory.ReadWrite.All is a tenant-wide key sitting on some server, and I don't want to be the one explaining that to a client. Where the credential livesUnder Certificates & secrets you can upload a certificate or create a client secret. I prefer certificates, because the private key stays in the machine's certificate store and never gets pasted into a file. If you do use a secret, give it a short expiry, put the renewal date on a calendar someone actually reads, and never hard-code it. We keep ours in a SecretStore vault on the box that runs the job: Install-Module Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore -Scope CurrentUser Register-SecretVault -Name LocalStore -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault Set-Secret -Name 'GraphReportApp' -Secret (Read-Host -AsSecureString 'Client secret')Run that as the same account the scheduled task runs as, because the vault is per-user. For unattended use you'll also need to set the SecretStore's password behaviour to suit your environment (see Set-SecretStoreConfiguration). Getting a token and following the pagesThe token endpoint is https://login.microsoftonline.com/<TenantID>/oauth2/v2.0/token. With client credentials the scope is always https://graph.microsoft.com/.default, which means "whatever application permissions have been consented". The response contains access_token, which is good for roughly an hour. Graph hands results back in pages. When there are more, the response includes an @odata.nextLink property with the full URL of the next page, and you keep following it until it's gone. If you skip this, the report quietly stops at the first page (100 users by default). Nobody notices for months, until someone asks why the licence count looks low. The full script# Get-GraphInventory.ps1 - list users and Intune managed devices via Microsoft Graph # Requires: Microsoft.PowerShell.SecretManagement + a registered vault holding the secret $TenantId = '<TenantID>' $ClientId = '<AppClientID>' $Secret = Get-Secret -Name 'GraphReportApp' -AsPlainText function Get-GraphToken { $body = @{ client_id = $ClientId scope = 'https://graph.microsoft.com/.default' client_secret = $Secret grant_type = 'client_credentials' } $tokenUri = "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" (Invoke-RestMethod -Method Post -Uri $tokenUri -Body $body ` -ContentType 'application/x-www-form-urlencoded').access_token } function Invoke-GraphPaged { param([Parameter(Mandatory)][string]$Uri, [Parameter(Mandatory)][hashtable]$Headers) $all = [System.Collections.Generic.List[object]]::new() do { $page = Invoke-RestMethod -Method Get -Uri $Uri -Headers $Headers foreach ($item in $page.value) { $all.Add($item) } $Uri = $page.'@odata.nextLink' } while ($Uri) $all } $headers = @{ Authorization = "Bearer $(Get-GraphToken)" } # Single quotes matter: in double quotes PowerShell would try to expand $select $users = Invoke-GraphPaged -Headers $headers ` -Uri 'https://graph.microsoft.com/v1.0/users?$select=displayName,userPrincipalName,accountEnabled&$top=999' $devices = Invoke-GraphPaged -Headers $headers ` -Uri 'https://graph.microsoft.com/v1.0/deviceManagement/managedDevices?$select=deviceName,operatingSystem,osVersion,lastSyncDateTime,userPrincipalName' "{0} users, {1} managed devices" -f $users.Count, $devices.Count $devices | Sort-Object lastSyncDateTime | Select-Object deviceName, operatingSystem, osVersion, lastSyncDateTime, userPrincipalName | Export-Csv -Path .\managed-devices.csv -NoTypeInformationIf you'd rather use a certificateDoing certificate auth with raw Invoke-RestMethod means building and signing a JWT client assertion yourself. It's doable but fiddly, so when we use certificates we let the Microsoft Graph PowerShell SDK do that part: # Certificate auth with the Graph SDK - no secret on disk at all Connect-MgGraph -TenantId '<TenantID>' -ClientId '<AppClientID>' -CertificateThumbprint '<Thumbprint>' Get-MgDeviceManagementManagedDevice -All | Select-Object DeviceName, OperatingSystem, LastSyncDateTimeIt's the same app registration with the same permissions. You just upload the public key (.cer) under Certificates & secrets. Checking the token before you trust the report $t = Get-GraphToken $t.Length # a long string, not an error ($t -split '\.')[1] # middle segment of the JWT - decode it to check the "roles" claimDecode the middle part of the token with a JWT decoder you trust, or base64-decode it locally. The roles claim should list User.Read.All and DeviceManagementManagedDevices.Read.All. If roles is missing, consent hasn't been granted.Compare the device count in the CSV with the count in the Intune admin center. If yours comes out at exactly 100, 999 or 1000, paging is broken.Things that have bitten us Double-quoted URLs. "...users?$select=..." expands $select to an empty string, and Graph happily returns every property. Use single quotes, or escape the dollar sign with a backtick.Long runs outlive the token. A big tenant can take more than an hour to page through, so on long jobs request a fresh token before the old one expires instead of reusing one for the whole run.Throttling. Push Graph too hard and it returns 429 with a Retry-After header. Wrap your calls in a retry that honours that header rather than hammering it.Expired secrets. This is the most common "the report stopped" ticket we get. The error is AADSTS7000222 (expired client secret). Track expiry dates somewhere you actually look.Adding the Delegated version of a permission looks right in the portal and does nothing for this flow.

Certificates expire on a schedule you know years ahead, and expired certificates are still one of the most common outages we get called about. My favorite example was a public VPN portal whose certificate had expired roughly six months before anyone told us. Users had just been clicking through the warning. Nothing was monitoring it, because it was an appliance, not a web server anyone thought of as a website. When we replaced the certificate, a second problem showed up. The appliance wasn't serving the intermediate certificate. Browsers papered over that, but stricter clients refused to connect. Here's how we check for both problems by hand and with a small script. Gather this first A Linux or macOS shell with opensslA list of every TLS endpoint you're responsible for, including websites, VPN portals, mail servers, firewall and appliance admin pages, and RDP gatewaysSomewhere to run a scheduled job and send its output, like email, a chat webhook or your monitoring systemSee what the server is really sendingConnect and show the certificates the server sends. Always pass -servername so SNI picks the right certificate on servers that host more than one name. Without it, a shared server may hand you its default certificate, and the check looks fine while the real site is broken. openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com -showcerts </dev/nullThe output lists each certificate the server sent, numbered from 0 (the leaf). Each one shows s: (subject) and i: (issuer). A healthy chain from a public CA usually has the leaf at 0 and one or more intermediates after it. If only certificate 0 appears and its issuer is an intermediate CA, the server isn't sending its chain. Reading the expiry datePipe the leaf into openssl x509 to read its dates: echo | openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com 2>/dev/null \ | openssl x509 -noout -subject -issuer -enddatesubject=CN = vpn.contoso.com issuer=C = US, O = Example CA, CN = Example Intermediate R1 notAfter=Mar 10 23:59:59 2027 GMTIn scripts, I lean on -checkend SECONDS. It exits non-zero if the certificate expires within that many seconds, so you don't have to parse dates or deal with the differences between GNU and BSD date. Checking the chainLook at the verification result at the end of the s_client output: openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com </dev/null 2>/dev/null | grep 'Verify return code'You want Verify return code: 0 (ok). The failures we see most: 21 (unable to verify the first certificate) or 20 (unable to get local issuer certificate) usually means the server is missing an intermediate.10 (certificate has expired) means what it says.18 or 19 means a self-signed certificate, or a private CA this machine doesn't trust. For internal hosts, pass -CAfile with your root.So why don't browsers complain about a missing intermediate? Many browsers cache intermediates or fetch them using the Authority Information Access URL in the leaf. Command-line tools, VPN clients, Java and Python applications, mail servers and older mobile devices often don't, so they fail while the browser looks fine. That's why I never test with a browser alone. The fix is to install the full chain on the server, meaning the leaf plus intermediates but not the root. The script we run every morningPut your endpoints in a text file, one host port per line: vpn.contoso.com 443 mail.contoso.com 443 portal.contoso.com 8443 fw01.contoso.com 443Then run this script from cron daily: #!/usr/bin/env bash # cert-check.sh - warn on certificates expiring soon or with broken chains # usage: cert-check.sh hosts.txt [warn_days] LIST="${1:-hosts.txt}" WARN_DAYS="${2:-30}" SECS=$((WARN_DAYS * 86400)) status=0 while read -r host port; do [[ -z "$host" || "$host" == \#* ]] && continue port="${port:-443}" out=$(timeout 15 openssl s_client -connect "$host:$port" -servername "$host" </dev/null 2>/dev/null) cert=$(printf '%s\n' "$out" | openssl x509 2>/dev/null) if [[ -z "$cert" ]]; then echo "ERROR $host:$port no certificate retrieved"; status=2; continue fi end=$(printf '%s\n' "$cert" | openssl x509 -noout -enddate | cut -d= -f2) verify=$(printf '%s\n' "$out" | grep -m1 'Verify return code' | sed 's/^ *//') if ! printf '%s\n' "$cert" | openssl x509 -noout -checkend 0 >/dev/null; then echo "EXPIRED $host:$port $end"; status=2 elif ! printf '%s\n' "$cert" | openssl x509 -noout -checkend "$SECS" >/dev/null; then echo "WARN $host:$port expires $end (under $WARN_DAYS days)"; status=1 else echo "OK $host:$port expires $end" fi if [[ "$verify" != *"0 (ok)"* ]]; then echo "CHAIN $host:$port $verify"; [[ $status -lt 1 ]] && status=1 fi done < "$LIST" exit $statusSchedule it so it only emails when the exit code is non-zero. That way, a certificate email in your inbox always means something needs attention. 0 7 * * * /opt/scripts/cert-check.sh /opt/scripts/hosts.txt 30 > /tmp/cert-check.out || mail -s 'Certificate check' noc@contoso.com < /tmp/cert-check.outMost monitoring systems can do this natively too, with HTTP checks on certificate age or a TLS sensor. If you already have one, use it. We still find the script handy for appliances on odd ports and as a second opinion. Make sure the check itself works Run the script against a host you know is fine and confirm it prints OK with the right date.Point it at a test site with a known-bad certificate (badssl.com has expired and incomplete-chain examples) and confirm you get EXPIRED and CHAIN lines and a non-zero exit code.Confirm the cron email actually arrives.Where certs slip past us Renewal isn't installation. Automated renewal on one box doesn't update the copy someone manually loaded onto a load balancer or appliance.Missing intermediates hide from browsers. Trust Verify return code over what the browser shows you.For us the hard part wasn't the script, it was building the list. VPN gateways, firewall admin pages, UPS cards and printers are the ones that expire unnoticed, so build the list from DNS and your firewall NAT rules rather than from memory.

Most of our small clients will never pay for an expensive SIEM, but they still deserve someone watching their logs. Wazuh is how we cover that. It's free and open source, and it gives a small MSP a lot of what the big products offer, across Windows, Linux and macOS. We run a single Wazuh manager for every client. This post covers how we deploy agents from the RMM, keep clients separated, cut down the noise, and keep the manager itself from turning into a liability. What we started with A Wazuh server (manager, indexer and dashboard). For a few hundred agents, the all-in-one install on a VM with 8 vCPU and 16 GB RAM or more is a good start.Agents that can reach the manager on TCP 1514 (events) and 1515 (enrollment)An RMM or script deployment tool for Windows and Linux endpointsWhat you get out of itLog collection and analysis. Agents ship Windows event logs, syslog and application logs to the manager, which decodes them and matches them against thousands of rules.File integrity monitoring (FIM), which alerts when watched files, directories or registry keys changeVulnerability detection. Agents inventory installed packages and the manager compares them against CVE feeds.Security configuration assessment (SCA), which scores endpoints against CIS-style benchmark checksActive response, which is optional automated action such as blocking an IP after repeated failures. I'd be careful with this one.Lock down enrollment firstOut of the box, anything that can reach port 1515 can enroll as an agent. Turn on password enrollment on the manager. In /var/ossec/etc/ossec.conf, inside the <auth> section, set <use_password>yes</use_password>, then write the password to the auth file: echo 'REPLACE-WITH-LONG-RANDOM-STRING' > /var/ossec/etc/authd.pass chmod 640 /var/ossec/etc/authd.pass chown root:wazuh /var/ossec/etc/authd.pass systemctl restart wazuh-managerOne group per clientGroups let you push different configuration to different agents, and they let whoever is looking at the dashboard filter down to one client. Create them on the manager before you deploy anything: /var/ossec/bin/agent_groups -a -g client-contoso -q /var/ossec/bin/agent_groups -a -g client-fabrikam -q /var/ossec/bin/agent_groups -lEach group gets a shared config file at /var/ossec/etc/shared/GROUP/agent.conf, and agents in that group pull it automatically. That's where client-specific FIM paths or extra log sources go. Pushing agents from the RMMThe agent packages accept deployment variables, so one command installs the agent, enrolls it and assigns the group. Match the agent version to your manager version or older, because an agent newer than the manager won't connect. On Windows: $msi = "$env:TEMP\wazuh-agent.msi" Invoke-WebRequest -Uri 'https://packages.wazuh.com/4.x/windows/wazuh-agent-4.x.x-1.msi' -OutFile $msi Start-Process msiexec.exe -Wait -ArgumentList "/i `"$msi`" /q WAZUH_MANAGER=`"wazuh.contoso.com`" WAZUH_REGISTRATION_PASSWORD=`"REPLACE-ME`" WAZUH_AGENT_GROUP=`"client-contoso`"" Start-Service -Name WazuhSvcReplace 4.x.x with the exact version you run. On Debian or Ubuntu, with the Wazuh repository already added: WAZUH_MANAGER='wazuh.contoso.com' WAZUH_REGISTRATION_PASSWORD='REPLACE-ME' \ WAZUH_AGENT_GROUP='client-contoso' apt-get install -y wazuh-agent systemctl daemon-reload systemctl enable --now wazuh-agentKeep the registration password in an RMM secret or script variable instead of hard-coding it in a script every technician can see. The first week is loudExpect a flood at first. Don't disable whole rule groups. Write targeted overrides in /var/ossec/etc/rules/local_rules.xml instead, where a child rule with level 0 suppresses a specific match. For example, this silences successful logons from a known service account: <group name="local,windows,"> <rule id="100100" level="0"> <if_sid>60106</if_sid> <field name="win.eventdata.targetUserName">^svc-backup$</field> <description>Ignore routine logons by the backup service account</description> </rule> </group>Use custom rule IDs from 100000 upward. Test by pasting a sample event into /var/ossec/bin/wazuh-logtest, then restart the manager. Treat the manager like a domain controllerYour SIEM holds logs from every client and can push configuration to every agent. If someone gets into it, they get into everyone. Don't expose the dashboard to the internet without SSO and MFA in front of it, or keep it reachable only over VPN.Change the default admin passwords for the dashboard, indexer and API right after install.If agents connect over the internet, expose only 1514 and 1515, and consider restricting them to client office IPs.Keep the API port (55000) internal.Back up /var/ossec/etc (keys, rules, groups), and plan indexer retention so the disk doesn't fill.Checking agents landed in the right place /var/ossec/bin/agent_control -l /var/ossec/bin/agent_groups -l -g client-contosoAgents should show as Active and appear in the right group. In the dashboard, filter by group and confirm events, SCA results and vulnerability data are coming in. Then create a test file in a monitored directory and watch for the FIM alert. Things that caught us out An agent newer than the manager won't connect. Upgrade the manager first, every time.Cloned VMs can carry duplicate agent keys. A template that already has an enrolled agent causes name and ID collisions, so remove the agent's client.keys before sealing a template.The indexer disk fills up. Set index lifecycle policies early.Vulnerability alerts pile up fast. We report vulnerabilities per client instead of alerting on each CVE.Each week I look at the top rules by count in the dashboard and tune whichever one is worst. It only takes a few minutes, and it's what keeps our techs reading the alerts instead of muting them.

LibreNMS is what we use to watch switches, routers, firewalls, UPSes and access points across our client sites. When a closet switch drops, we want the alert to reach whoever is on call before the office staff come in and find it. LibreNMS auto-discovers a lot over SNMP, graphs ports and health sensors out of the box, and can alert to chat, ticketing and paging systems. This is how we set up SNMPv3 on a device, add it, write alert rules and wire up transports. I've also included a few transport bugs that cost us alerts before we caught them. What to have ready A working LibreNMS install. We run it in a container, but a native install is the same from here on.Network reachability from the LibreNMS poller to the device on UDP 161Admin access to the deviceWebhook or API details for wherever alerts should go, such as Discord, your ticket system or PagerDutySNMPv3 on the deviceUse SNMPv3 with authentication and privacy (authPriv). SNMPv2c community strings travel in clear text. Here's a generic Cisco IOS example with a read-only view and an ACL limiting who can poll: ip access-list standard SNMP-POLLERS permit 10.10.20.5 ! snmp-server view LIBRENMS-VIEW iso included snmp-server group LIBRENMS-GRP v3 priv read LIBRENMS-VIEW access SNMP-POLLERS snmp-server user librenms LIBRENMS-GRP v3 auth sha AUTH-PASSPHRASE priv aes 128 PRIV-PASSPHRASE snmp-server location Main Office MDF snmp-server contact noc@contoso.comOn IOS, SNMPv3 users don't show up in the running config, so put the credentials in your password manager right away. You can check the user with show snmp user. It's also worth setting a meaningful location string, because LibreNMS uses it for maps and grouping. Prove SNMP works before LibreNMS gets involvedI always test with snmpwalk from the LibreNMS host before adding a device. It tells you whether you have an SNMP problem or a LibreNMS problem: snmpwalk -v3 -l authPriv -u librenms -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1 sysDescrIf that returns the device description, add it from the web UI (Devices, Add Device) or the CLI: cd /opt/librenms sudo -u librenms ./lnms device:add --v3 -l authPriv -u librenms \ -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1Discovery runs, and the device fills in ports, sensors and hardware details over the next few polling cycles. A short list of alert rulesLibreNMS ships a set of default rules you can import from Alerts, Alert Rules. Start small. An alert stream nobody reads is worse than none. These are the rules we always deploy: Device down, meaning device status is 0 with ICMP and SNMP both failingPort down on uplinks, scoped to ports with a description you tag rather than every access portSensor over limit, for temperature, PSU and UPS battery stateDevice rebooted, meaning uptime below a few minutes, which catches power eventsUse device groups, for example one per client or site, to scope rules and route them to the right transport. Set a sensible delay so a single missed poll doesn't page anyone. Where the alerts goTransports are what deliver alerts. Under Alerts, Alert Transports, add one per destination: For Discord, paste the channel webhook URL. It's good for team visibility.Most ticket systems can create tickets by email, or through an API transport that POSTs JSON to an endpoint.For PagerDuty, use the integration key from a PagerDuty service. We only page for real outages, so nobody gets woken up for a warning.Map rules to transports directly on the rule or through transport groups. Press the Test button on each transport before you rely on it. Proving it end to end Run the validator and fix anything it flags: cd /opt/librenms sudo -u librenms ./validate.phpThe device shows recent poll times and the graphs are filling in.Each transport's Test button delivers a message.Unplug a lab device or shut a test port, and confirm the alert fires, appears in the alert log, and clears when you put it back.The bugs that cost us alerts On the version we were running, an API transport saved without its options field populated (even as empty) threw a PHP error when an alert fired. It didn't just skip that transport, it stopped the whole alert run. If alerts stop arriving everywhere at once, go looking for one broken transport.A PagerDuty transport created without the region field set caused the same kind of error and killed the run. Open each PagerDuty transport, set the region explicitly, and save it.If you write your own webhook receiver, look at the actual JSON. Fields like alert state arrived as strings in our payloads, and a receiver comparing them to integers silently dropped every alert.There's no automatic retry. If a transport endpoint is down when the alert fires, that notification may simply be lost, so monitor your receivers too.The alert log in the UI shows what fired. When something fired but never arrived, the answer is usually in /opt/librenms/logs/librenms.log.If you add a second poller, add it to the device ACL too. Otherwise it fails with timeouts that look a lot like wrong credentials.After any change to transports, I press Test on every one of them. One broken transport can silence all the others, and you won't find out until a real outage goes unreported.

This started as a routine look at a client's remote access portal logs. We found around 50,000 password-spray sessions a day coming from just five networks, with zero successful logins. Nobody had noticed, because spraying is quiet by design. Instead of hammering one account with thousands of passwords, the attacker tries a few common passwords against thousands of usernames, which stays under per-account lockout thresholds. Anything with a public login page gets this as constant background noise. Here's how we spotted it, measured it and shut it off at the edge. Access you'll want The portal or VPN authentication logs, whether that's local logs, a syslog server or a SIEMA Linux shell with the usual tools (grep, awk, sort, uniq)Admin access to the edge firewall or load balancer in front of the portalWhat it looks like in the logsOnce you've seen it, the pattern is easy to recognize: Thousands to tens of thousands of authentication sessions per day, far more than your user count could explainLots of usernames, often ones that don't exist in your directory, like admin, test, scanner, or first.last guesses from a scraped staff listA handful of source /24 networks, often hosting providers, rotating through addresses within each blockNo successes at all. If any attempts succeed, stop reading and start incident response.Counting failures by networkEvery product logs differently, so you'll need to adapt the field extraction. Here's a generic log line: 2026-10-01T03:12:44Z portal auth=fail user=jsmith src=203.0.113.45 reason=bad_passwordCount failed attempts per source /24: grep 'auth=fail' portal.log \ | grep -oE 'src=[0-9.]+' | cut -d= -f2 \ | awk -F. '{print $1"."$2"."$3".0/24"}' \ | sort | uniq -c | sort -rn | head -20Then check how many distinct usernames each network tried. A real user behind a NAT fails with one or two names. A sprayer tries hundreds. grep 'auth=fail' portal.log \ | awk '{for(i=1;i<=NF;i++){if($i~/^user=/)u=substr($i,6); if($i~/^src=/)s=substr($i,5)} split(s,o,"."); print o[1]"."o[2]"."o[3]".0/24", u}' \ | sort -u | awk '{print $1}' | uniq -c | sort -rn | head -20And confirm there were no successes from those networks: grep 'auth=success' portal.log | grep -E 'src=(203\.0\.113|198\.51\.100)\.' | wc -lFor this client, five networks accounted for essentially all of the failures, each had tried hundreds of usernames, and the success count was zero. Block it before it reaches login and MFAWhere you block matters. If the portal forwards each attempt to your MFA provider or directory, the spray costs you more than log space. It loads your domain controllers, can lock out real staff whose usernames happen to match, and can spam people with push prompts if a password ever does match. Drop the traffic before it gets to the login logic. Most platforms have a named list you can block against. It might be an address list or address group on a firewall, a data group referenced by a load balancer rule, or an external dynamic list. Keep the list in one place so adding a network is a one-line change. On a Linux edge, ipset does the same job: ipset create spray-block hash:net ipset add spray-block 203.0.113.0/24 ipset add spray-block 198.51.100.0/24 iptables -I INPUT -p tcp --dport 443 -m set --match-set spray-block src -j DROPMake the rule persistent using your distribution's mechanism, and put a comment or ticket reference on each entry. Whoever looks at that list next will want to know why it's there. Geo-blocking, if your users are all in one countryBlocking five networks handles today's sprayer, and the next one will come from somewhere else. If every legitimate user is in one country, allowing only that country to reach the portal removes most of the noise with one rule. Most commercial firewalls have built-in geo objects. Keep an exception process for staff who travel. Geo-IP data is approximate and VPN providers can appear to be anywhere, so I treat this as noise reduction, not a security control on its own. MFA and lockout still do the real workEdge blocking cuts load and noise, but it doesn't protect accounts. Attackers rotate sources, and spraying is slow on purpose. What actually protects your users: MFA on every remote access path, preferably phishing-resistant or number-matching, so a guessed password alone isn't enoughA lockout or smart-lockout policy that slows guessing without making it trivial to lock out the whole staffBanned-password lists so seasonal passwords like Fall2026! get rejectedAlerts on any successful login from an unusual country or network, and on MFA prompts users report they didn't triggerNext-day checks Re-run the per-network count on the next day's log. The blocked networks should be gone.The firewall rule's hit counter should be climbing.Real users can still log in, including from home ISPs and mobile carriers.The MFA provider logs no longer show a stream of authentication requests for unknown users.Be careful with these Before blocking a network, check that it isn't a mobile carrier or a large ISP your users share. Hosting provider ranges are usually safe to block. Residential ranges are not.If a load balancer sits in front of the portal, make sure the real client IP is being logged. Otherwise every attempt looks like it came from one internal address.A block rule placed below a broad allow rule never matches. Check the hit counts after you add it.The list is easy to forget about. Review the blocked networks every few months and remove stale entries.We re-run the per-network count on that portal every so often. New sprayers keep turning up from new places, and adding another /24 to the list takes about a minute.

When we inherit a Proxmox cluster, backups are one of the first things I look at. Proxmox makes them easy to turn on, which also makes it easy to assume you're covered when you aren't. This is what we run for our own cluster and for clients: which vzdump mode we pick, how we schedule and prune, why I think Proxmox Backup Server is worth the extra box, and the restore testing that tells you whether any of it works. Have these ready A Proxmox VE node or clusterBackup storage that is not on the same disks or pool your VMs run on. An NFS share or a separate disk works, and a Proxmox Backup Server (PBS) is better.A spare VM ID range for restore tests. We use 9000 and up.A snapshot isn't a backupA Proxmox snapshot is a point-in-time state stored on the same storage as the VM. It's perfect for "I'm about to upgrade this and want to roll back in five minutes if it goes badly." It does nothing for you if the storage fails, the pool fills, someone deletes the VM, or ransomware reaches the host. Long-lived snapshots also slow down some storage types and eat space as the disk changes. Take them before a change and delete them after. Backups are separate copies on separate storage. Picking a vzdump modeSnapshot mode keeps the guest running while it's backed up. For VMs, Proxmox uses QEMU's live backup mechanism, and with the QEMU guest agent installed and enabled it briefly freezes the guest filesystems for consistency. For containers it needs storage that supports snapshots, or it falls back to suspend mode. This is our default.Suspend mode, for containers, copies data while running and then suspends the container briefly for a final sync. For VMs it offers little over snapshot mode and causes a pause.Stop mode shuts the guest down, backs it up and starts it again. It's the most consistent, at the cost of downtime. We use it for a few legacy systems whose databases don't tolerate live backup well.Install and enable the QEMU guest agent on every VM: qemu-guest-agent on Linux, the VirtIO guest tools on Windows, and the Guest Agent option in the VM settings. Without it, snapshot-mode backups are crash-consistent, not filesystem-consistent. A one-off manual backup looks like this: vzdump 101 --mode snapshot --storage backup-nfs --compress zstd --notes-template '{{guestname}} manual'Scheduling, and why hand-picked lists go wrongCreate jobs under Datacenter, Backup in the GUI. They're stored cluster-wide in /etc/pve/jobs.cfg and run on whichever node holds each guest. We usually run one nightly job for everything, outside business hours, selecting by pool or by "all" with exclusions instead of a hand-picked list of VM IDs. Hand-picked lists are how a new VM ends up never backed up. Point the job's failure notifications at someone who actually reads that inbox. A backup job that's been failing silently for three weeks is the most common problem we find on inherited clusters. How long to keep thingsRetention is set with prune options on the job or on the storage. For most small environments we start here: keep-last=3, keep-daily=7, keep-weekly=4, keep-monthly=6On the command line the same thing is: vzdump 101 --storage backup-nfs --prune-backups keep-daily=7,keep-weekly=4,keep-monthly=6That gives you a week of dailies, a month of weeklies and half a year of monthlies. The monthlies are what save you when someone notices that data went missing back in the spring. Proxmox Backup Server is worth the extra boxBacking up to a plain NFS share works, but every backup is a full compressed archive. PBS changes that in a few ways: Data is stored in chunks and identical chunks are stored once, so nightly backups of mostly unchanged VMs take a small fraction of the space.With dirty bitmaps, a running VM only sends changed blocks after the first backup, so jobs finish much faster.Verify jobs re-read chunks and check their checksums on a schedule, which catches silent corruption before you need the backup.Sync jobs to a second PBS offsite give you the offsite copy in a 3-2-1 setup.You also get file-level restore from the GUI, encryption, and per-user permissions.On PBS, schedule prune, garbage collection and verify jobs per datastore. Prune only marks old snapshots for removal. Garbage collection is what actually frees the space. Restore to a new ID and actually boot itA backup you've never restored is a hope, not a backup. Restore to a new VM ID so the production guest is left alone: # VM from a vzdump archive on file storage qmrestore /mnt/pve/backup-nfs/dump/vzdump-qemu-101-2026_10_01-02_00_03.vma.zst 9101 --storage local-lvm --unique # VM from PBS (volume ID as shown by pvesm list) pvesm list pbs01 --vmid 101 qmrestore pbs01:backup/vm/101/2026-10-01T09:00:03Z 9101 --storage local-lvm --unique # Container pct restore 9201 /mnt/pve/backup-nfs/dump/vzdump-lxc-201-2026_10_01-02_10_44.tar.zst --storage local-lvmBefore you boot the copy, take it off the network. Otherwise it can fight the production guest for its IP or start sending email. qm set 9101 --net0 virtio,bridge=vmbr0,link_down=1 qm start 9101When you rewrite net0, keep the NIC model and bridge that qm config 9101 shows. Then open the console and confirm the OS boots, services start and the data is current. Delete the test guest when you're done. What a healthy setup looks like The backup job's task log shows every expected guest with status OK.The storage view lists backups at the retention points you expect.On PBS, the last verify job is green.You have a dated note of the last successful restore test for each important system.Mistakes we keep running into Backups sitting on the same pool as the VMs. If Ceph or the RAID dies, both go together. Backup storage has to be independent.Restoring over the original by accident. Restoring to the existing VM ID overwrites it, so always use a new ID unless you mean to replace production.Duplicate MAC and IP. Without --unique and a disconnected NIC, a restored VM can knock the original off the network.Trusting snapshot mode for databases. A filesystem freeze isn't the same as an application-consistent database backup, so run native dumps for critical databases as well.A full PBS datastore. If garbage collection isn't running, pruned backups still take up space until the datastore fills.We restore something every month. If the dated note for an important system is more than a month old, that restore is the next thing on my list.

Over a few years we'd piled up a fleet of cloud VPS instances: a monitoring box, a ticketing box, a couple of web hosts, a mail relay, and a scattering of one-off servers. Each was cheap on its own, but together they cost us about $334 a month. We consolidated everything onto our own Proxmox cluster in a colocation rack, and the hosting bill dropped to about $112 a month. That's roughly two thirds less, and it keeps heading toward near zero as the last stragglers move. Honestly, the bigger win for us was having one place to back up, monitor and patch. Here's the method that let us do it without the people we support seeing any downtime. What we had in place first A Proxmox cluster with enough capacity and a backup targetRoutable public addresses at the new location. We use a small public block.A router or firewall VM in front of the cluster. We use VyOS, and OPNsense or pfSense work too.Control of DNS for every hostname being movedA spreadsheet. Seriously.Write down what every box really runsBefore moving anything, log into every VPS and write down what it actually runs, not what you remember it running. For each box we captured: Listening ports and the processes behind them (ss -tlnup)Every hostname pointing at it, from DNS and from web server configsCron jobs and systemd timers for every userDatabases, where their data lives, and how big they areInbound dependencies like webhooks, firewall allowlists at third parties, and API clients that connect by IPOutbound dependencies like SMTP relays and API keys bound to source IPsThe cron jobs and IP allowlists are what get you. A forgotten nightly report, or a vendor that only accepts connections from your old IP, won't show up until a week after cutover when someone asks where their report went. One container per serviceI get the urge to rebuild the old all-in-one boxes as they were. Resist it. We gave each service its own LXC container (or a VM where it needed one) with its own IP on an internal services VLAN, for example 10.10.20.0/24. Now each service gets backed up, restored, upgraded and moved on its own, and the next migration becomes moving one small thing at a time. You own the firewall nowYour VPS provider was giving you a firewall for free, even if you never thought about it. Now it's your job. We run a VyOS VM with the public uplink on one side and the internal VLANs on the other. The policy is simple. Inbound is dropped by default, established and related return traffic is allowed, and only the specific ports each published service needs are opened. Internal VLANs can reach the internet, but the internet reaches nothing unless a rule says so. A single reverse proxy for the web stuffMost of what we moved was HTTP. Instead of giving every web app a public IP, we run one reverse proxy container and forward ports 80 and 443 to it. Ours is Nginx Proxy Manager, though Caddy, Traefik or plain Nginx are just as good. It terminates TLS, holds the certificates, and routes by hostname to the right internal container, so one public IP can serve dozens of sites. 1:1 NAT for the few that need their own IPSome things don't fit behind a shared HTTP proxy. Think mail servers that need matching reverse DNS, VPN endpoints, game or voice servers on fixed ports, and anything a third party allowlists by IP. For those we use 1:1 NAT on the router, mapping a dedicated public address to one internal container, with firewall rules limited to the ports that service uses. Keep this list short, because every 1:1 mapping is one more exposed surface to watch. Test the new side while DNS still points at the old oneThis is the step that made our cutovers boring, which is exactly what you want. Copy the data, bring the service up on the new side, and test it while DNS still points at the old server. Lower the DNS TTL to 300 seconds a day or two ahead so the eventual flip propagates quickly. From inside the network, hit the backend directly with a Host header: curl -sI -H 'Host: app.contoso.com' http://10.10.20.15/Then test the full public path, including the router, reverse proxy and TLS certificate, without changing DNS. --resolve pins the hostname to the new public IP: curl -sv --resolve app.contoso.com:443:192.0.2.10 https://app.contoso.com/ -o /dev/null 2>&1 | grep -E 'HTTP/|subject:|expire'You can do the same thing in a browser by temporarily adding the name to your hosts file. Log in, click around, submit a form, and check that background jobs run. Only once it works end to end do we touch DNS. Flip DNS, but keep a way backFor services with a database, do a final sync in a quiet window. Stop the app on the old side, take a last dump, restore it on the new side, then flip DNS. After that: Stop the old VPS. Don't delete it. A stopped instance can be started again in a minute if something was missed.Take a provider snapshot of the old box as a second rollback and keep it until you're confident. Snapshots usually cost a fraction of a running instance.If you can, watch the old box's logs for a few days before stopping it. Any traffic still arriving points you at a hard-coded IP or a forgotten client.Signs the move actually stuck dig +short app.contoso.com returns the new address from several resolvers.The certificate on the new proxy is valid and auto-renewing.Cron jobs, scheduled reports and outbound email are firing from the new location.Each new container shows up in your backup jobs and monitoring.Things that surprised us Several integrations called our old server by IP rather than hostname. The access logs on the box we kept running a little longer are how we found them.Apps behind the proxy suddenly see every request coming from the proxy's IP, which breaks rate limiting and fail2ban. Configure trusted proxies and X-Forwarded-For handling in each app.A new public IP has no sending history and probably no matching PTR record. Fix reverse DNS and SPF before moving anything that sends mail.Don't cancel the VPS account first. Cancel the instances, keep the snapshots, and close the account only once you no longer need them.Moving one small service at a time felt slow while we were doing it. Looking back, it's the reason nobody had to call us about it.

Someone wants to try a self-hosted tool, and of course it ships as a Docker image. A whole VM for it feels like overkill, so most of the time we run Docker inside an LXC container on Proxmox instead. Containers start in seconds, share the host kernel and use far less RAM. It works well once the right features are turned on. Here's how we set it up, how we give a container a TUN device so Tailscale or WireGuard can run inside it, and when I give up and build a VM. Before you start A Proxmox VE host. We're on the 8.x and 9.x series.A Debian or Ubuntu container template downloaded to local storageRoot shell access to the Proxmox hostStay unprivilegedIn an unprivileged container, root inside the container maps to a high, unprivileged UID on the host (UID 100000 by default). If something breaks out, it lands on the host as a nobody user. In a privileged container, root inside is root on the host, so a container escape is a host compromise. Privileged mode does avoid some friction, like bind-mounted host directories with odd ownership, certain device access, or NFS mounts inside the container. We still treat it as a last resort. If an app only works in privileged mode, that usually tells me it belongs in a VM. When I build a VM insteadThe workload needs its own kernel, kernel modules or a different kernel version.You need live migration without downtime. Containers restart when they migrate.The service is internet-facing and untrusted, and you want the stronger isolation boundary.The stack leans on Docker features that fight with nesting, or it's Kubernetes.It's Windows, or anything else that isn't Linux.For a handful of internal services, Docker in LXC is fine, and we run plenty of them. Creating the containerTwo features matter. nesting=1 lets the container create its own nested namespaces and cgroups, which Docker needs. keyctl=1 allows the keyctl system call in unprivileged containers. Docker and runc use it, and without it you get failures that are confusing to track down. pveam update pveam available --section system | grep debian-12 pveam download local debian-12-standard_12.7-1_amd64.tar.zst pct create 210 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst \ --hostname docker01 \ --unprivileged 1 \ --features nesting=1,keyctl=1 \ --cores 2 --memory 2048 --swap 512 \ --rootfs local-lvm:16 \ --net0 name=eth0,bridge=vmbr0,ip=10.10.20.21/24,gw=10.10.20.1 \ --onboot 1 pct start 210Use the template filename that pveam available actually lists, because the version suffix changes over time. For a container you already have, turn the features on under Options, Features in the GUI, or run pct set 210 --features nesting=1,keyctl=1 and restart it. Installing DockerGet in with pct enter 210 and install Docker from Docker's official repository using their documented Debian steps, or use the distribution's docker.io package if you prefer. Then run a test container: docker run --rm hello-world docker info | grep -i 'storage driver'You want the overlay2 storage driver. If you see vfs, every image layer is a full copy and disk use balloons. That used to be common on ZFS-backed container storage. Newer ZFS releases support overlayfs, so check what your host is running. Giving it /dev/net/tunTailscale and WireGuard userspace implementations need /dev/net/tun, and unprivileged containers don't get it by default. Stop the container and add two lines to its config file on the host: pct stop 210 nano /etc/pve/lxc/210.conflxc.cgroup2.devices.allow: c 10:200 rwm lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=fileThe first line allows access to the character device with major 10, minor 200, which is TUN. The second bind-mounts the host's device node into the container and creates the file if it doesn't exist. Start the container again and install Tailscale the normal way: pct start 210 pct enter 210 ls -l /dev/net/tun curl -fsSL https://tailscale.com/install.sh | sh tailscale upIf the container is going to be a subnet router or exit node, also enable IP forwarding inside it (net.ipv4.ip_forward=1 in a sysctl file) and approve the routes in the Tailscale admin console. That last bit is the part I always forget. Quick sanity checks docker run --rm hello-world finishes without permission errors.docker info shows overlay2.Inside the container, ls -l /dev/net/tun shows a character device, and tailscale status shows the node online.On the host, pct config 210 shows unprivileged: 1 and the features line.Where this has bitten us Bind-mounted host folders show up as nobody:nogroup. That's UID mapping doing its job. Either chown the host directory to the mapped UID (container UID plus 100000) or set up a custom idmap. Please don't switch to privileged just to fix permissions.Updates can break nesting. We've seen Docker, containerd or runc updates inside containers start failing with AppArmor or permission errors until the host side caught up. Take a pct snapshot before upgrading Docker in any container that matters.vzdump snapshot mode doesn't quiesce databases running in Docker. For anything with a database, run its native dump before the backup or use stop mode.Moving the container to another node restarts it, along with every Docker service inside.I try to keep it to one service per container. Once a single container is running twenty stacks, you've rebuilt a VM with weaker isolation, and that's usually my cue to move it to an actual VM.

When the support contract ran out on a 4-node Nutanix block, the easy move would have been to recycle it. I'm glad we didn't. Those chassis are close to ideal Proxmox VE and Ceph hardware: identical nodes, SSD and HDD in every sled, dual 10GbE, and IPMI on each one. Ours now runs a good chunk of our internal workload. This is the order we built it in, plus what I'd plan differently next time. Have these lined up first The block itself. We used four nodes, and three is the minimum for a sane Ceph cluster.IPMI access to each node for console and virtual mediaThe Proxmox VE ISO, on USB or mounted over IPMIA switch with VLAN support, ideally 10GbE for CephAn IP plan for each network below, and working NTPNetworks first, because moving Ceph later hurtsChanging Ceph networks after the fact is painful, so we settled them before installing anything. We use four logical networks, each on its own VLAN: Management (for example 10.10.10.0/24) carries the Proxmox web UI, SSH and corosync. Corosync is latency-sensitive, so keep it off any link Ceph is saturating. A second corosync link on another network is cheap insurance.Ceph public (10.10.30.0/24) is where monitors and clients talk to OSDs.Ceph cluster (10.10.40.0/24) carries OSD-to-OSD replication and recovery. This is the link that gets hammered when a disk fails.VM traffic rides a VLAN-aware bridge carrying the guest networks.On a two-port 10GbE node, we put Ceph public and cluster on one port, management and VM traffic on the other, and used the 1GbE ports as a corosync backup. Disks: Proxmox on the boot device, everything else to CephNutanix nodes usually boot from a small SATADOM or M.2 device and keep the data drives behind an HBA. Install Proxmox on the boot device and leave every data disk for Ceph. Give each data disk its own OSD and don't put RAID underneath it, since Ceph is your redundancy layer. Keep SSDs and HDDs in separate pools with separate CRUSH rules. If you mix them in one pool, the slowest disk sets the pace for everything. And set expectations with whoever is planning capacity: with the default 3-way replication, usable space is roughly a third of raw. Install every node, then ping everythingInstall from the ISO on each node with a unique hostname and a static management IP. After first boot, set up your repositories, update, and build the bridges and VLAN interfaces in /etc/network/interfaces or the GUI. Before you go further, make sure every node can ping every other node on the management, Ceph public and Ceph cluster networks. A VLAN typo is a lot easier to find now than after Ceph is running on top of it. Building the clusterOn the first node: pvecm create hx-clusterThen join the rest using the first node's management IP: pvecm add 10.10.10.11 pvecm statuspvecm status should list every node and show Quorate: Yes. I join nodes one at a time and let each one settle before adding the next. Ceph packages, monitors and OSDsInstall the Ceph packages on every node, then initialize the config once with both networks: # on every node pveceph install --repository no-subscription # once, on the first node pveceph init --network 10.10.30.0/24 --cluster-network 10.10.40.0/24Monitors need a majority to keep quorum, so run an odd number of them. On four nodes that means three. Run these on three different nodes: pveceph mon create pveceph mgr createThen create one OSD per data disk on each node. Ceph assigns the device class (hdd, ssd or nvme) by itself, and you can check it later with ceph osd tree. lsblk -o NAME,SIZE,ROTA,MODEL pveceph osd create /dev/sdb pveceph osd create /dev/sdc # ...repeat for every data disk on every nodeIf a disk refuses to become an OSD Disks from the previous platform still carry its metadata, and old partitions block OSD creation. Wipe them first with ceph-volume lvm zap /dev/sdX --destroy. Triple-check the device name before you hit enter. Pools split by disk typeCreate one replicated rule per device class, then a pool for each. The --add_storages flag registers the pool as Proxmox storage across the cluster. ceph osd crush rule create-replicated rule-ssd default host ssd ceph osd crush rule create-replicated rule-hdd default host hdd pveceph pool create vm-ssd --crush_rule rule-ssd --add_storages pveceph pool create vm-hdd --crush_rule rule-hdd --add_storagesLeave the defaults of size=3 and min_size=2. We put VM boot disks and databases on the SSD pool and bulk data on the HDD pool. Watch the order here. If you create a pool before your class-specific rules exist, it uses the default rule that spans every disk. You can change it later with ceph osd pool set POOL crush_rule rule-ssd, but that triggers data movement, so it's easier to set the rule when you create the pool. CephFS for the ISO pileRBD pools hold disk images, but ISOs, container templates and backups-in-transit need a filesystem. Put a metadata server on at least two nodes for failover, then create the filesystem: pveceph mds create # on two or more nodes pveceph fs create --name cephfs --add-storageEvery node now sees the same ISO and template library. It's a small thing, but it's one of my favorite quality-of-life wins over local storage. Before you trust it with real VMs ceph -s ceph osd tree ceph osd df pvesm statusYou want HEALTH_OK, all monitors in quorum, every OSD up and in, and placement groups active+clean. Then break it on purpose. Migrate a VM between nodes, then reboot one node and watch ceph -s go degraded and recover on its own. A few more things I'd tell the next personUpdate BMC and BIOS firmware while the nodes are still empty, and confirm the HBA presents disks directly to the OS.Point chrony at a reliable NTP source on every node. Monitors complain about clock skew if the node clocks drift.Don't fill it up. Ceph gets unhappy as OSDs approach the nearfull ratio. We plan to stay well under 80 percent so a node failure can re-replicate without hitting full.If I were doing it again, I'd spend even longer on the network plan, since that's the one decision here that's genuinely painful to undo.

Most of the self-hosted apps we look after are a database plus a folder of uploads and config. You don't need a backup suite for that. A short shell script, cron and a second location do the job, as long as the database dump is consistent, retention is sensible and a copy gets off the box. Then you test the restore, because the day someone calls about missing files is a bad day to find out your backups were empty. You'll need the host running the app with MySQL or MariaDB, mysqldump and gzip (in the MySQL/MariaDB client package), and somewhere off-box. That can be another server you can rsync to over SSH (192.0.2.10 in the examples) or an object storage bucket with rclone configured. Keep the password off the command line A password on the command line shows up in ps and shell history. Put it in /root/.backup.my.cnf instead: [client] user=backup password=long-random-string chmod 600 /root/.backup.my.cnf Give that MySQL user only what a dump needs, which is SELECT, SHOW VIEW, TRIGGER, LOCK TABLES and PROCESS. MySQL 8 wants PROCESS for tablespace information, or you can use --no-tablespaces. The script Save this as /usr/local/sbin/backup-myapp.sh: #!/usr/bin/env bash set -euo pipefail APP="myapp" DB_NAME="myapp" APP_DIR="/var/www/myapp" BACKUP_DIR="/var/backups/${APP}" RETENTION_DAYS=14 REMOTE="backup@192.0.2.10:/srv/backups/${APP}/" STAMP="$(date +%F_%H%M)" umask 077 mkdir -p "${BACKUP_DIR}" DB_FILE="${BACKUP_DIR}/${APP}_db_${STAMP}.sql.gz" FILES_FILE="${BACKUP_DIR}/${APP}_files_${STAMP}.tar.gz" # Remove partial files if anything fails cleanup() { rm -f "${DB_FILE}.tmp" "${FILES_FILE}.tmp"; } trap cleanup EXIT echo "$(date -Is) starting backup of ${APP}" # 1. Database: consistent snapshot without locking InnoDB tables mysqldump --defaults-extra-file=/root/.backup.my.cnf \ --single-transaction --quick --routines --triggers --no-tablespaces \ "${DB_NAME}" | gzip > "${DB_FILE}.tmp" gzip -t "${DB_FILE}.tmp" mv "${DB_FILE}.tmp" "${DB_FILE}" # 2. Uploads and config tar -czf "${FILES_FILE}.tmp" -C "${APP_DIR}" uploads config mv "${FILES_FILE}.tmp" "${FILES_FILE}" # 3. Local retention find "${BACKUP_DIR}" -type f -name "${APP}_*" -mtime +"${RETENTION_DAYS}" -delete # 4. Off-box copy rsync -a "${BACKUP_DIR}/" "${REMOTE}" echo "$(date -Is) backup of ${APP} complete" chmod 700 /usr/local/sbin/backup-myapp.sh /usr/local/sbin/backup-myapp.sh ls -lh /var/backups/myapp/ A few parts of that script are there because they've burned us before. set -euo pipefail stops the script on any error, including a failed mysqldump in the middle of a pipe. Without pipefail, a failed dump piped into a successful gzip counts as success, and you get a tiny, useless file every night. The 20-byte backups Before we added pipefail, a client's backups ran "successfully" for weeks after a password change broke mysqldump. Each night produced a valid gzip of an empty stream. A size check or one restore test would have caught it on day one. --single-transaction takes a consistent snapshot of InnoDB tables without locking them, so people can keep using the app during the dump. It doesn't protect MyISAM tables, so if your app still uses those, check before relying on it. --quick streams rows instead of buffering whole tables in memory. Writing to .tmp and renaming means a failed run never leaves behind a file that looks like a good backup, and gzip -t checks the archive is intact. The find -mtime +14 -delete line removes local copies older than 14 days, matched by name so it can't wander into other files. rsync -a without --delete means the remote keeps files even after local retention removes them, so give the remote its own retention job. That longer remote history matters more than it sounds. If backups have been failing for longer than your local retention window, local cleanup can leave you with nothing usable, and the remote is all you've got. Getting a copy off the box For rsync, set up an SSH key for root to the backup host and restrict it on that end. For object storage, swap the last step for rclone: rclone copy "${BACKUP_DIR}" "b2:example-backups/${APP}" --max-age 48h On-box backups cover you for a bad update or an accidental delete. They do nothing for a dead disk, a deleted VM, a compromised host or ransomware, because those take the backups along with the data. I like a pull-based setup even better, where the backup server connects in and fetches the files, so a compromised app server has no credentials that can delete the backup history. Object storage with versioning or object lock gets you something similar. Scheduling Create /etc/cron.d/backup-myapp: 30 2 * * * root /usr/local/sbin/backup-myapp.sh >> /var/log/backup-myapp.log 2>&1 Files in /etc/cron.d need the user field (root here), unlike a personal crontab. Pick a quiet time and stagger it from other heavy jobs. Cron won't tell anyone when a job fails. Have the script ping a dead man's switch service (healthchecks-style) on success, so you get alerted when the ping doesn't arrive. Prove it restores Do this when you set it up, and on a schedule after that (we do quarterly). Restore into a scratch database, never the live one: LATEST=$(ls -1t /var/backups/myapp/myapp_db_*.sql.gz | head -n1) mysql -u root -p -e "CREATE DATABASE restore_test;" gunzip -c "$LATEST" | mysql -u root -p restore_test mysql -u root -p -e "SELECT table_name, table_rows FROM information_schema.tables WHERE table_schema='restore_test';" mysql -u root -p -e "DROP DATABASE restore_test;" mkdir -p /tmp/restore-test tar -xzf "$(ls -1t /var/backups/myapp/myapp_files_*.tar.gz | head -n1)" -C /tmp/restore-test ls -R /tmp/restore-test | head rm -rf /tmp/restore-test Because the dump was taken of a single database without --databases, it has no CREATE DATABASE or USE statements, so it restores cleanly into whatever database name you pick. Compare the table list and rough row counts with production. The best test is a full one, restoring both parts onto a spare VM and logging in to the app. Also check the off-box copy actually exists by listing it from the backup host, not by trusting the log.

When someone hands us a self-hosted app running in Docker, the trouble is almost never Docker itself. It's a .env full of passwords anyone on the box can read, data living somewhere nobody can point to, an update that quietly pulled a breaking major version, or backups nobody ever took. Compose fits this kind of work well because one YAML file describes the whole stack, so we use the same layout and routine on every Compose host. That way whichever tech gets the call already knows where everything is. You'll need Docker Engine with the Compose plugin (docker compose, not the old docker-compose), and root or sudo. One directory per app under /opt /opt/wiki/ compose.yaml .env data/ # bind-mounted application data db/ # bind-mounted database data Everything about the app lives in one place, so backing it up, moving it or rebuilding the host means dealing with one directory. Compose also uses the directory name as the project name, so containers and networks get predictable names. Secrets go in .env, locked down DB_PASSWORD=long-random-string APP_SECRET=another-long-random-string chmod 600 /opt/wiki/.env chown root:root /opt/wiki/.env Compose reads .env from the project directory automatically for variable substitution, so compose.yaml can reference ${DB_PASSWORD} and still be safe to share or commit. Never commit the .env itself. The compose file services: db: image: postgres:16 restart: unless-stopped environment: POSTGRES_DB: wiki POSTGRES_USER: wiki POSTGRES_PASSWORD: ${DB_PASSWORD} volumes: - ./db:/var/lib/postgresql/data app: image: ghcr.io/example/wiki:2.5 restart: unless-stopped depends_on: - db environment: DB_HOST: db DB_PASSWORD: ${DB_PASSWORD} APP_SECRET: ${APP_SECRET} ports: - "127.0.0.1:8080:3000" volumes: - ./data:/app/data A few things in there are on purpose. unless-stopped brings containers back after a crash or reboot but respects a deliberate docker compose stop. With no restart policy, nothing comes back after a reboot, and you find out when someone calls. The image is pinned to :2.5 rather than :latest, so an update is a decision and not a surprise. I bump the tag after reading the release notes. The app is published on 127.0.0.1 only because a reverse proxy on the same host fronts it. Docker's port publishing bypasses typical host firewall rules, so don't count on those. The app reaches the database as db on the project's default network, which means the database port never gets published at all. Bind mounts or named volumes? For self-hosting I prefer bind mounts (./data:/app/data) inside the project directory. The data sits in a directory you chose, you can see it, back it up with ordinary tools, and it moves with the project. Named volumes are managed by Docker under /var/lib/docker/volumes/ and avoid some file-ownership headaches, but they're easier to lose track of, and docker compose down -v deletes them. If your data is in named volumes, that's your data gone. Bind mounts aren't touched. We only use named volumes where an image's documentation strongly recommends them. The cost of bind mounts is permissions. Containers that run as a non-root user need the host directory owned by that UID. Check the image docs, then chown -R the host directory to match before first start. Day to day cd /opt/wiki docker compose up -d # create or update and start everything docker compose ps # status docker compose logs -f --tail=100 app docker compose restart app docker compose down # stop and remove containers (data stays) Updating without surprises cd /opt/wiki # 1. back up first (Step 6) # 2. change the image tag in compose.yaml if moving versions docker compose pull docker compose up -d docker compose logs -f --tail=100 docker image prune -f pull fetches new images and up -d recreates only the containers whose image or configuration changed. Watch the logs until migrations finish before you call it done. Backups For a consistent copy, stop the stack, archive the directory and start it again: cd /opt docker compose -f /opt/wiki/compose.yaml stop tar -czf /var/backups/wiki-$(date +%F).tar.gz wiki docker compose -f /opt/wiki/compose.yaml start If you can't take the downtime, dump the database with its own tool instead (for example docker compose exec -T db pg_dump -U wiki wiki) and archive the data directory alongside it. Either way, copy the result off the host. Cap the logs before they fill the disk The default json-file log driver has no size limit. Set one in /etc/docker/daemon.json: { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } Restart Docker afterwards. It only applies to newly created containers, so run docker compose up -d --force-recreate in each project to pick it up. Don't docker cp your config in We once needed to customise a config file inside a running container, so we used docker cp to drop it in and restarted. The change disappeared, and on another image the container wouldn't start at all. Many images run an entrypoint script on every start that does a recursive chown of the data directory, regenerates config from environment variables, or runs under set -e so any unexpected state aborts startup. Whatever you copied in gets overwritten, has its ownership changed, or trips the script. And it's lost entirely the next time an update recreates the container. Put the file in the project directory and bind-mount it instead, read-only where you can: volumes: - ./data:/app/data - ./config/custom.ini:/app/config/custom.ini:ro Now the change is part of the project, survives updates and gets backed up with everything else. Before I call a host done docker compose ps shows all services running (and healthy, if they define health checks) Reboot the host and everything comes back without anyone touching it ls -l /opt/wiki/.env shows -rw------- A backup archive restored to a test host comes up with docker compose up -d If you only have time for one of those, do the restore test. It's the one that tells you whether the rest of this was worth it.

This setup came out of a PHP app's web installer telling me No such file or directory when I pointed it at the database. We'd moved MySQL into a container because upgrades and rollbacks are much easier than with the distro package, but the app itself ran on the host under nginx and PHP-FPM. Apps like that usually expect MySQL on the local Unix socket at /var/run/mysqld/mysqld.sock, the same as a native install. So this is how we do it now, with TCP on loopback only and the socket shared to the host. Why localhost gave a file error We'd only published the TCP port, and the installer had the database host set to localhost. With mysqli, the word localhost doesn't mean TCP at all. It tries the default Unix socket (/var/run/mysqld/mysqld.sock on Debian builds), and that socket didn't exist because MySQL was only listening on TCP inside a container. The error sounds like a missing file in the app, which sent me looking in the wrong place. You can fix it by entering 127.0.0.1 as the host to force TCP, or by sharing the socket as below so localhost works the way the app expects. We went with the socket, so the app config looks exactly like a native install. You'll need Docker Engine with the Compose plugin, and root or sudo. Make sure no MySQL or MariaDB server package is installed on the host, because it will fight over port 3306 and /run/mysqld. Client packages are fine. Project directory and root password mkdir -p /opt/mysql/data cd /opt/mysql Put the root password in /opt/mysql/.env and lock it down: MYSQL_ROOT_PASSWORD=change-me-to-a-long-random-string chmod 600 /opt/mysql/.env A socket directory that survives reboots /var/run is a symlink to /run, which is a tmpfs, so anything you create there is gone after a reboot. A tmpfiles.d entry recreates it at boot with the right owner, before Docker starts the container. Without it, Docker creates the missing bind-mount source itself as a root-owned directory, and depending on the image and timing, mysqld may not be able to create its socket there. I'd rather not leave that to luck. First check the UID of the mysql user inside the image: docker run --rm mysql:8.4 id mysql In the official image that's UID and GID 999. Create /etc/tmpfiles.d/mysqld-docker.conf: d /run/mysqld 0755 999 999 - The 0755 matters. The socket file itself is world-writable, but the host user (www-data, for example) has to be able to enter the directory, or the app gets "Permission denied" on the socket. Apply it now without rebooting: systemd-tmpfiles --create /etc/tmpfiles.d/mysqld-docker.conf ls -ld /run/mysqld The compose file /opt/mysql/compose.yaml: services: mysql: image: mysql:8.4 container_name: mysql restart: unless-stopped env_file: .env ports: - "127.0.0.1:3306:3306" volumes: - ./data:/var/lib/mysql - /var/run/mysqld:/var/run/mysqld 127.0.0.1:3306:3306 publishes the port on loopback only. Leave the IP prefix off and Docker publishes on all interfaces, and Docker's own firewall rules are applied ahead of most host firewall setups, so a ufw "deny 3306" won't save you. Bind-mounting /var/run/mysqld puts the container's socket directory on the host. When mysqld creates mysqld.sock inside the container, it shows up on the host at the same path. Start it and check the socket docker compose up -d docker compose logs -f mysql # wait for "ready for connections", then Ctrl+C ls -l /var/run/mysqld/ You should see mysqld.sock. Confirm the server agrees on the path: docker exec -it mysql mysql -uroot -p -e "SHOW VARIABLES LIKE 'socket';" If your image version reports the socket somewhere else, change the bind mount to match that directory, or set the socket path explicitly in a config file mounted into /etc/mysql/conf.d/. The app's database and user docker exec -it mysql mysql -uroot -p CREATE DATABASE appdb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER 'appuser'@'localhost' IDENTIFIED BY 'another-long-random-string'; GRANT ALL PRIVILEGES ON appdb.* TO 'appuser'@'localhost'; Socket users and TCP users aren't the same MySQL sees connections over the Unix socket as coming from localhost. Connections through the published TCP port arrive from the Docker bridge network instead, so a 'appuser'@'localhost' account won't match them. If the app will use TCP, create the user for the bridge subnet (or %, since the port is only on loopback). This is the part I always forget. Making sure it holds up From the host, with only the client package installed: apt install -y default-mysql-client mysql -u appuser -p appdb -e "SELECT CURRENT_USER(), @@hostname;" mysql -h 127.0.0.1 -u root -p -e "SELECT 1;" ss -ltnp | grep 3306 The first mysql command uses the socket (no host given), the second uses TCP. ss should show 3306 bound to 127.0.0.1 only. Then reboot the host and check that /run/mysqld exists, the container is up and the socket is back.

Most of the small web apps we host live the same way. One LXC container or VM per service on Proxmox, nginx and PHP-FPM inside it, and a separate reverse proxy in front handling TLS. Each app is isolated, backups and rebuilds are easy, and when a PHP upgrade goes sideways only one app is down instead of everybody's. Two details get missed almost every time. The app needs to see the real client IP instead of the proxy's, and PHP needs to know the original request was HTTPS. Miss those and you get logs full of one IP, broken rate limiting and redirect loops. What I'm assuming you have A Debian 12 LXC or VM for the app, 10.10.20.21 in the examples A reverse proxy (nginx, Nginx Proxy Manager, Caddy, Traefik) at 10.10.20.5, handling TLS A DNS name pointed at the proxy Install nginx and PHP-FPM apt update apt install -y nginx php-fpm php-mysql php-xml php-mbstring php-curl php-zip systemctl enable --now nginx php8.2-fpm ls /run/php/ On Debian 12 the default PHP is 8.2 and the FPM socket is /run/php/php8.2-fpm.sock. Use whatever the ls actually shows you. The proxy side The proxy has to pass along the original host, client IP and scheme. In plain nginx on the proxy: location / { proxy_pass http://10.10.20.21; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } Most GUI proxies, Nginx Proxy Manager included, set these headers for you. The app's nginx site Create /etc/nginx/sites-available/app.conf: # Translate the proxy's scheme header into an on/off value for PHP map $http_x_forwarded_proto $https_flag { default off; https on; } server { listen 80; server_name app.example.com; root /var/www/app/public; index index.php index.html; # Trust X-Forwarded-For only from our reverse proxy set_real_ip_from 10.10.20.5; real_ip_header X-Forwarded-For; real_ip_recursive on; client_max_body_size 64m; location / { try_files $uri $uri/ /index.php?$query_string; } location ~ \.php$ { include snippets/fastcgi-php.conf; fastcgi_pass unix:/run/php/php8.2-fpm.sock; fastcgi_param HTTPS $https_flag; } location ~ /\.(?!well-known) { deny all; } } The map block sits outside server because it has to be in the http context. Debian includes site files inside http, so that's fine. set_real_ip_from plus real_ip_header make nginx replace $remote_addr with the client IP from the header, but only when the request comes from the trusted proxy. Anyone else sending a forged header gets ignored. fastcgi_param HTTPS sets $_SERVER['HTTPS'], which is what most PHP apps and frameworks check when building URLs and deciding whether to redirect. Debian's fastcgi.conf (included by the snippet) also sends HTTPS $https if_not_empty, but $https is empty on a plain HTTP listener, so only our value is sent. If the app has a "force HTTPS" setting and this flag isn't passed, it sees every request as plain HTTP and redirects forever. Some frameworks also need their own trusted proxy setting (Laravel's TrustProxies, for example). A lot of guides online tell you to add try_files $uri =404; inside the PHP location. On Debian and Ubuntu, snippets/fastcgi-php.conf already has a try_files line, so adding yours makes nginx -t fail with "try_files" directive is duplicate. Leave it out. The snippet already returns 404 for PHP files that don't exist. Enable the site and remove the default one: ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/ rm /etc/nginx/sites-enabled/default nginx -t && systemctl restart nginx The reload that looked like it worked If you skip nginx -t and run systemctl reload nginx with a broken config, the reload fails but nginx keeps running the old configuration. The site stays up, so it looks like nothing happened, and you lose time wondering why your change has no effect. Worse, the next full restart (or a reboot) fails and the site goes down. My habit is always nginx -t, then restart, then actually test with a browser or curl -I. Lock down the backend Only the proxy should be able to reach port 80 on the app container. If clients can hit it directly, they can bypass the proxy and send whatever X-Forwarded-Proto they like. Use the Proxmox firewall, nftables or your network firewall to allow TCP 80 from 10.10.20.5 only. One more thing that bites the first time someone tries to upload a big file. client_max_body_size has to be raised on both the proxy and the backend, and PHP's upload_max_filesize and post_max_size in /etc/php/8.2/fpm/php.ini need to match. If they don't, uploads fail with a 413. Restart php8.2-fpm after changing them. Checking it end to end Drop a temporary test file in the web root: cat > /var/www/app/public/ipcheck.php <<'EOF' <?php header('Content-Type: text/plain'); echo 'REMOTE_ADDR: ' . $_SERVER['REMOTE_ADDR'] . "\n"; echo 'HTTPS: ' . ($_SERVER['HTTPS'] ?? '(unset)') . "\n"; EOF Browse to https://app.example.com/ipcheck.php. You should see your own public IP and HTTPS: on. Check that tail -f /var/log/nginx/access.log shows real client addresses, not 10.10.20.5. Then delete the test file so it isn't sitting there for anyone to find.

Our help desk resets passwords and unlocks accounts all day long. When we take over a small or mid-sized domain, the most common over-privilege I find is the help desk (or the office manager who does it on the side) sitting in Domain Admins just so they can do that. AD has had granular delegation since Windows 2000, and it takes a few minutes to set up. Further down is a related trick I use from the RMM a lot: getting a machine, and scripts running as SYSTEM on it, to pick up a new group membership without rebooting someone's PC in the middle of their day. Delegating resets and unlocks You'll need Domain Admin once to set it up, users organised into OUs so you can scope it, and a security group for the people getting the rights, for example GRP-Helpdesk-PwReset. Always delegate to a group, never to individuals. When someone joins or leaves the help desk you change group membership, not ACLs. In Active Directory Users and Computers, right-click the OU containing the users (for example OU=Staff,DC=contoso,DC=local) and choose Delegate Control. Add GRP-Helpdesk-PwReset. Under common tasks, tick Reset user passwords and force password change at next logon. Finish the wizard. That common task doesn't include unlocking, which catches people out. Run the wizard again on the same OU, choose Create a custom task to delegate, then Only the following objects in the folder, tick User objects, and on the permissions page show Property-specific permissions and tick Read lockoutTime and Write lockoutTime. If you're setting this up across a lot of OUs or clients, I'd script it with dsacls instead: $ou = "OU=Staff,DC=contoso,DC=local" $grp = "CONTOSO\GRP-Helpdesk-PwReset" dsacls $ou /I:S /G "${grp}:CA;Reset Password;user" dsacls $ou /I:S /G "${grp}:RPWP;pwdLastSet;user" dsacls $ou /I:S /G "${grp}:RPWP;lockoutTime;user" /I:S applies the entry to child objects only, CA is a control access right (Reset Password), and RPWP is read and write property. The help desk can then use ADUC from RSAT, or PowerShell: Set-ADAccountPassword -Identity jdoe -Reset -NewPassword (Read-Host -AsSecureString "New password") Set-ADUser -Identity jdoe -ChangePasswordAtLogon $true Unlock-ADAccount -Identity jdoe Why it won't work on your admins Members of protected groups (Domain Admins, Account Operators and so on) have adminCount=1, and their permissions get reset hourly from the AdminSDHolder object. Your help desk delegation won't apply to them, by design. Honestly, that's what you want. The help desk shouldn't be able to reset a Domain Admin's password. New group membership for SYSTEM, no reboot Computer accounts get group memberships too, usually to grant access to a file share, a certificate template or a GPO's security filter. The catch is that a computer gets its Kerberos tickets when it boots and renews them for hours afterwards. Add it to a group and it won't "see" the membership until it gets a new ticket, which normally means a reboot. That's a hard sell when someone is halfway through their work on that machine. You can force it by purging the tickets for the SYSTEM logon session (LUID 0x3e7). The next time anything running as SYSTEM needs a ticket, the machine asks the DC for a fresh TGT, and that one carries the new group. First add the computer to the group. Note the trailing $ on the computer's account name. Add-ADGroupMember -Identity "GRP-Deploy-Share-Read" -Members "WS-0142$" Make sure the change has replicated to the DC the machine authenticates against (repadmin /syncall /APed if you're in a hurry). Then on the target machine, from an elevated prompt or your RMM, which usually runs as SYSTEM anyway: klist -li 0x3e7 purge gpupdate /force # if the group is used for GPO security filtering certutil -pulse # if the group grants enrollment on a certificate template Now the script running as SYSTEM can reach the share or service that's restricted to the group. Access the resource by its DNS name, not its IP address, so Kerberos actually gets used. If the membership was only for a one-off job, like copying installers from a restricted share or a single enrollment, take the computer back out when it's done and purge again so the access goes away now and not in ten hours: Remove-ADGroupMember -Identity "GRP-Deploy-Share-Read" -Members "WS-0142$" -Confirm:$false # then on WS-0142: klist -li 0x3e7 purge Put the removal in the same script or ticket as the add. We've found computers sitting in "temporary" groups months after the job, because nobody came back for them. Where the trick stops working Purging tickets refreshes what the machine presents to other servers over Kerberos. It doesn't rebuild the local token of processes already running as SYSTEM, so whoami /groups in an existing SYSTEM session won't change, and neither will local permission checks against the new group. Anything that depends on the local token still needs a reboot. Testing both halves Log on as a help desk member (not an admin) and reset a test user's password and unlock them. Try the same on a user outside the OU. It should be denied. After the purge, klist -li 0x3e7 shows a fresh TGT issued a moment ago, and the SYSTEM-context task can read the restricted resource. I mostly use the purge for remote one-off jobs where a reboot would interrupt someone, for testing GPO filtering or certificate autoenrollment without rebooting each time, and for temporary access during a deployment script. It isn't a substitute for proper standing access. If machines need a group permanently, put them in it and let them pick it up on their next reboot like normal.

The VPN, the remote access portal, the copier that does scan-to-email, the wiki, the helpdesk system. Anything that looks people up in AD needs an account to bind with, and in a lot of environments we inherit, that account belongs to an actual person. Usually whoever set it up. It works fine until they change their password at the 90-day prompt, mistype it on their phone and lock themselves out, or leave. Then everybody's logins to that system break at once. It's messy in quieter ways too. The logs show that person "logging in" thousands of times a day, and because those accounts are often admins, the appliance ends up holding admin credentials. What we do instead is boring on purpose. One dedicated service account per system, minimum rights, a long random password, no interactive logon, and a written rotation process. The one we caught right before a handover Before handing a client environment over to another provider, we audited every integration and found the remote access portal binding to AD as one of our own engineers, by name. The day that account got disabled, nobody at the client would have been able to connect remotely. We created a dedicated read-only bind account, tested it with ldapsearch, switched the portal over, confirmed staff could log in, and added it to the handover documentation. Ten minutes of work, and the new team didn't spend their first morning on a remote access outage. To do this you'll need rights to create OUs, users and GPOs, a password manager or vault for the generated password, and access to the config of whatever system is going to bind. Give service accounts their own OU Keeping them out of your normal user OUs means user GPOs, password policies and cleanup scripts don't touch them by accident. New-ADOrganizationalUnit -Name "Service Accounts" -Path "OU=Admin,DC=contoso,DC=local" -ProtectedFromAccidentalDeletion $true Create the account Generate the password in your password manager first. We go 32 characters or more. Name the account so whoever finds it in two years knows what it's for, and put the owner and runbook in the description. $pw = Read-Host "Password for svc-vpn-ldap" -AsSecureString New-ADUser -Name "svc-vpn-ldap" ` -SamAccountName "svc-vpn-ldap" ` -UserPrincipalName "svc-vpn-ldap@contoso.local" ` -Path "OU=Service Accounts,OU=Admin,DC=contoso,DC=local" ` -AccountPassword $pw ` -Enabled $true ` -PasswordNeverExpires $true ` -CannotChangePassword $true ` -Description "LDAP bind for remote access portal. Owner: IT. See runbook SVC-001." "Password never expires" is a deliberate trade-off. An expiring password on a service account causes exactly the outage we're trying to avoid. The compensating controls are the long random password and the rotation process further down. Keep it to Domain Users For read-only lookups (authenticate a user, read their group memberships), plain Domain Users membership is enough, since authenticated users can read most user and group attributes by default. Please don't add it to Domain Admins or any admin group "to make it work". If an app genuinely needs to write, like a password self-service tool, delegate that specific permission on the specific OU and nothing else. Block interactive logon A bind account should never be able to sit at a desktop or open an RDP session. Make a group, something like GRP-Deny-Interactive-Logon, and add the account. Then in a GPO linked at the domain level (or the OUs holding your servers and workstations), go to Computer Configuration, Policies, Windows Settings, Security Settings, Local Policies, User Rights Assignment and add the group to Deny log on locally and Deny log on through Remote Desktop Services. Don't add it to Deny access to this computer from the network on your DCs. LDAP binds are network logons, so that breaks the one thing the account exists to do. Use TLS, and test before you touch the appliance Set the appliance to LDAPS (636) or StartTLS if it supports it. A simple bind over plain 389 sends the password in clear text. Before I change anything on the device, I test the bind from any Linux box with the OpenLDAP client tools: ldapsearch -H ldaps://dc01.contoso.local -D "svc-vpn-ldap@contoso.local" -W \ -b "DC=contoso,DC=local" "(sAMAccountName=jdoe)" memberOf If that returns the user's groups, the account works. Then update the appliance, test a real user's login, and only then remove the old credentials. Before you disable the old shared account, watch the DC logs for its logon attempts for a few days. There's always a second device nobody remembered, and in my experience it's usually a scan-to-email printer. How to tell it worked Users can log in to the system that binds to AD Get-ADUser svc-vpn-ldap -Properties LastLogonTimestamp, PasswordNeverExpires, MemberOf shows recent activity and only Domain Users Trying to RDP or log on locally as the service account is refused The old named account no longer shows bind activity in the DC security logs (look for its logons, events 4624 and 4625, from the appliance's IP) Write it down so rotation isn't scary For each service account we record what uses it and where the credential is entered (device, admin URL, config file), who owns the integration, where the password is stored, and when it was last rotated along with the steps. Rotation then is just generating a new password, updating the vault, setting it in AD, updating each consumer and testing. One account per system keeps that list short, and one rotation can't break five things. People sometimes ask why not use a gMSA. They're great for Windows services, but the password is retrieved by Windows hosts, so a VPN appliance or printer can't use one. For appliances, a normal user account managed like this is the right tool.

Time problems never show up as "the clock is wrong". They show up as a user who randomly can't authenticate, Group Policy that stops applying, or me trying to line up firewall logs against server logs and finding them minutes apart. Kerberos tolerates five minutes of clock skew by default, and past that things get weird fast. The design is simple once you see it. The DC holding the PDC emulator role in the forest root domain syncs to a reliable external NTP source. Every other DC syncs from the PDC emulator, or another DC higher up the hierarchy. Member servers and workstations sync from a DC in their domain, and that part happens on its own for domain-joined machines. So nearly all the work is on the PDC. You'll need admin access to your DCs, outbound UDP 123 from the PDC emulator to the NTP servers you pick, and about fifteen minutes. Find the PDC and see what it's doing netdom query pdc # or (Get-ADDomain).PDCEmulator Then on the PDC emulator: w32tm /query /status w32tm /query /source w32tm /query /configuration Read the Source and Stratum lines. If Source says Free-running System Clock, the server has no time source at all and just trusts its own clock. Local CMOS Clock is the same problem, it's reading the hardware clock. VM IC Time Synchronization Provider means it's getting time from the Hyper-V host, which I'll come back to. The domain that drifted together On one domain we took over, users were hitting intermittent Kerberos failures and the timestamps on firewall and server logs disagreed by several minutes. w32tm /query /source on the PDC came back Free-running System Clock. Nobody had ever given it an external source, so it was counting on its own oscillator, and every other machine in the domain was faithfully following it. The fix was exactly the next section, then a resync on the other DCs. The clients corrected themselves over the next few polling cycles. Point the PDC at external NTP w32tm /config /manualpeerlist:"time.nist.gov,0x8 0.pool.ntp.org,0x8 1.pool.ntp.org,0x8" /syncfromflags:manual /reliable:yes /update net stop w32time net start w32time w32tm /resync /rediscover /manualpeerlist is a space-separated list of peers. The ,0x8 suffix tells Windows to use client mode with them, which is what public NTP servers expect. /syncfromflags:manual says to use that list instead of the domain hierarchy. /reliable:yes advertises this DC as a reliable time source to the rest of the domain, and /update tells the service to load the new config. I always list more than one peer, so a single unreachable server doesn't leave you free-running again. If the status still shows the local clock afterwards, check that UDP 123 is actually getting out. w32tm /stripchart /computer:time.nist.gov timing out is a good sign the firewall is blocking it. And if the PDC is already far off, Windows may refuse to jump the clock because of the maximum correction limits (MaxPosPhaseCorrection and MaxNegPhaseCorrection). Set the time by hand close to correct, then resync. Put everyone else back on the hierarchy Run this on every other DC, and on any member server someone "fixed" by hand at some point: w32tm /config /syncfromflags:domhier /update net stop w32time net start w32time w32tm /resync w32tm /query /source The source should now be a DC name, not an internet server or the local clock. Virtual DCs shouldn't trust the host A virtual DC shouldn't take its time from the hypervisor. The host's clock is often the thing that's wrong, and then the whole domain is following an unmanaged box nobody ever set up NTP on. On Hyper-V, the cleanest fix that keeps the integration service's other benefits is to disable only the VM IC time provider inside the guest: reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t REG_DWORD /d 0 /f net stop w32time net start w32time w32tm /query /source You can also untick Time synchronization under Integration Services in the VM settings. On VMware, make sure periodic time sync with the host is off in VMware Tools for DC guests, and give the ESXi hosts proper NTP so any one-off syncs (after a vMotion or snapshot revert, for example) are at least correct. Checking your work w32tm /query /status w32tm /monitor w32tm /stripchart /computer:time.nist.gov /samples:5 /dataonly On the PDC, /query /status shows one of your NTP peers as the source, a sensible stratum (typically 2 to 4), and a recent Last Successful Sync Time. w32tm /monitor lists every DC with a small offset from the PDC. /stripchart against an external server shows offsets in milliseconds, not seconds. On a workstation, w32tm /query /source shows a DC. If you ever move the PDC role The manual peer config lives on the server, not the role. Transfer the PDC emulator to a new DC and the new one keeps following the hierarchy, so nothing in the domain talks to an external source anymore. Configure the new PDC as above and set the old one back to domhier. Some teams handle this with a GPO linked to the Domain Controllers OU, using a WMI filter that matches only the PDC (Select * from Win32_ComputerSystem where DomainRole = 5), so the setting follows the role automatically. If you've got ten seconds today, run w32tm /query /source on your PDC. I'm still surprised how often the answer is wrong.

A lot of the AD calls we get start the same way. Someone at the front desk says a user "only sometimes" can't get into an app, or a new hire's drive mappings never show up. When we dig in, it's rarely anything exotic. It's replication that quietly stopped weeks ago, a DC nobody noticed was heading for tombstone, or SYSVOL that never finished moving to DFSR. I run this same check on every domain we take on, and again any time something is flaky. You'll want a DC or a management box with RSAT (AD DS tools), an account in Domain Admins (or delegated rights to read replication metadata), and an elevated prompt or PowerShell window. Start with dcdiag across the whole enterprise dcdiag runs a battery of tests against a DC: connectivity, replication, advertising, FSMO reachability, SYSVOL and more. I always run it verbose and save it to a file, because the lines that matter scroll past way too fast on screen. dcdiag /v /c /e > C:\Temp\dcdiag-all.txt dcdiag /test:dns /v /e > C:\Temp\dcdiag-dns.txt /e tests every DC in the enterprise and /c runs the comprehensive set. Don't skip /e. Without it you're only testing the box you're logged into. We had one domain where the local DC was spotless and a branch office DC hadn't replicated in 40 days. Search the file for failed test. Some failures are noise. SystemLog, for example, fails if there's been any error in the System log in the last hour. Anything in Replications, Advertising, NetLogons, KccEvent or DFSREvent deserves a proper look. Then the replication summary repadmin /replsummary You get one table showing, per DC, the largest replication delta and how many of its partners are failing. You want zero in the fails column and deltas in minutes, not days. If any delta is getting close to your tombstone lifetime (180 days on modern forests), stop and fix that first. A DC that's been out of replication longer than that must not just be reconnected. Per-partner detail when something looks off repadmin /showrepl repadmin /showrepl * /csv > C:\Temp\showrepl.csv repadmin /queue /showrepl lists each naming context (Domain, Configuration, Schema, DomainDnsZones, ForestDnsZones) and each inbound partner with the result of its last attempt. Once you have more than a handful of DCs, the CSV is much easier to filter. /queue shows pending inbound requests. If the queue never drains, that DC is overloaded or can't reach its partner. The errors I see most here are 1722 (RPC server unavailable, usually firewall or DNS), 8453 (access denied on replication) and 8524 (DSA operation unable to proceed because of a DNS lookup failure). I check DNS first every time. Most replication errors turn out to be DNS errors in disguise. Forcing replication repadmin /syncall /APed The flags are A for all partitions, P to push changes outward from this DC, e to include partners in other sites, and d to identify servers by distinguished name in the output. Run it on the DC where the change was made. Every partition should report "SyncAll terminated with no errors". Then rerun repadmin /replsummary to confirm. The help desk ticket that was really replication This comes up more than you'd think. The help desk added a user to an access group on DC01 and could see it in ADUC, but the line-of-business app kept denying them. The app did its own LDAP lookups against DC02, which hadn't received the change yet. Within a site, changes normally replicate within seconds. Between sites the default interval is 180 minutes, and in this case someone had stretched the site link schedule further. A repadmin /syncall /APed on DC01 fixed it on the spot. So before you start troubleshooting the app, find out which DC it's actually talking to. nltest /dsgetdc:contoso.local on a Windows client works, or look at the app's LDAP server setting. Also remember that a user's own Windows session won't see a new group until they log off and back on, because group SIDs are baked into the logon token. That's a different problem from replication, so rule it out separately. FSMO roles netdom query fsmo You should get five roles, all held by DCs that actually exist and are online. If one points at a DC that was removed without being demoted, you'll need to seize that role to a healthy DC with ntdsutil and clean up the dead DC's metadata. While you're here, note which DC is the PDC emulator. It matters for time sync and password changes. Speaking of dead DCs, leftover DNS records and connection objects from DCs that were simply switched off throw replication errors that look alarming but are really a metadata cleanup job. Remove the dead DC in AD Sites and Services and clean up its DNS records instead of chasing each error. SYSVOL and DFSR dfsrmig /getglobalstate net share Get-CimInstance -Namespace root\microsoftdfs -ClassName DfsrReplicatedFolderInfo | Select-Object ReplicatedFolderName, State dfsrmig /getglobalstate should say Eliminated, which means the migration from FRS to DFSR is complete. Anything else means you're still on FRS, or halfway off it, and newer Windows Server versions won't accept that as a DC. net share has to show both SYSVOL and NETLOGON. A DC missing those won't process Group Policy for clients, and that's usually how users find out for you. In the CIM query, a State of 4 means Normal, 2 is initial sync and 5 is in error. Also check the DFS Replication event log on each DC for event 2213, which means replication paused after a dirty shutdown. It needs a manual resume, and it's the one I'm most likely to forget. What a healthy domain looks like dcdiag /e shows no failures in the replication, advertising or DFSR tests repadmin /replsummary shows zero fails and small deltas for every DC netdom query fsmo lists only live DCs Every DC shares SYSVOL and NETLOGON, and DFSR state is 4 I run this monthly and keep the output files. Comparing this month's against last month's is how you catch the slow drift before it turns into a Monday morning outage.