Keaton

Hey, and welcome to TechBytes. This started as our own notes: the fixes, the configs and the "never doing that again" moments from supporting cities, small businesses and our own servers. We figured they'd be more useful out in the open. A good place to start is Tutorials or Show & Tell. If you've got a minute, say hi in Introductions and tell us what you work on. And if you're stuck on something, post in Help & Q&A. That's what it's there for. Glad you're here. Keaton

One of the small cities we support has a rugged laptop in each police vehicle, connected through an in-car cellular router. The router brings up a tunnel back to the city network so officers can reach dispatch, records systems and file shares from the car. The tunnel was up, and pinging internal servers by IP worked. Even so, the laptops couldn't resolve anything in the city's internal Active Directory domain. Group Policy failed, domain authentication fell back to cached credentials, and any application that referred to a server by name broke. Officers had been working around it for a while, and every fix tried so far had been on the laptops themselves, like hosts files, manual DNS settings and re-joining the domain. None of it stuck. Asking each DNS server directlyFrom a vehicle laptop we ran the same lookup twice. The first went to the DNS server the laptop had been given by DHCP, and the second went straight to a domain controller's address across the tunnel. Querying the domain controller directly worked perfectly. Domain records, SRV records for the DCs, all of it.Querying the address handed out by DHCP returned NXDOMAIN for anything in the internal .local zone.The DHCP-assigned DNS server was the cellular router itself. Its built-in DNS forwarder was answering internal queries on its own instead of passing them up the tunnel. Two lookups, side by side, pointed straight at it. Why the router said noThe routers had a local domain setting used for naming DHCP clients, and its configured value overlapped the city's internal .local zone. As best we could tell, that made the router's forwarder treat itself as authoritative for those names. Anything not in its own lease table got an immediate NXDOMAIN, and the query never reached the domain controllers. The laptops trusted the answer, and Windows caches negative responses, which made the symptoms even less consistent from one car to the next. What we changedWe changed the routers' local domain setting fleet-wide through the vendor's cloud management platform, to a neutral value that doesn't overlap the AD domain. Every vehicle router was in the same configuration group, so this was one change pushed to the whole fleet instead of a visit to every car.We added AD sites and subnets for the vehicle address ranges. Without them, the domain controllers couldn't map the laptops to a site, so DC location was unpredictable and a laptop could pick a DC that was slow or unreachable over the tunnel.We created reverse lookup zones for the vehicle subnets, so the laptops could register PTR records and tools that rely on reverse lookups behaved properly.Rollout notes Negative caching made it look like the fix hadn't worked. After the router change, some laptops still failed because they'd cached the NXDOMAIN answers. Flushing the DNS client cache, or just rebooting, cleared it. We put that in the rollout notes so nobody would decide the fix was a dud.A few vehicles were parked with the router powered off, so not every router picked up the change at once. They took the new configuration when they next came online. We tracked the stragglers from the management portal instead of assuming we'd hit 100%.We almost fixed the wrong layer. Before the side-by-side test, the plan was to push static DNS servers to the laptops through Group Policy. That would have worked on the tunnel and failed badly as soon as a laptop was on any other network. Fixing the router kept DHCP as the single source of truth.Where it standsThe vehicle laptops resolve internal names reliably over LTE now. Group Policy applies in the field, and domain sign-in works instead of falling back to cached credentials. Domain controllers associate the laptops with the right site, so DC selection is predictable. The fix lives in one place, the router configuration group, so new vehicles get it automatically. A couple of things I'll carry forward. Small and mobile routers often run a local forwarder with features like local domains and rebind protection that can get in the way of internal zones, especially .local. Any time a client on the same router platform has odd DNS problems, that setting is the first thing I check. And whenever a new address range shows up, mobile and VPN ranges included, it goes into Sites and Services the day it's created, not the day something breaks.

We wanted somewhere to share what we learn on the job. That means how-tos, project write-ups, and the gotchas that cost us an afternoon, so hopefully they don't cost you one. A static blog felt too one-way, and a chat server buries everything within a week. We wanted a proper forum with search, tags and accounts, which led us to Invision Community. It had to run on our own infrastructure, be easy to rebuild, look like ours, and be backed up properly. What it runs onA Debian 13 LXC container on our Proxmox cluster. It's one service per container, same as everything else we host.nginx and PHP 8.4 (PHP-FPM) from the Debian packages.MySQL 8.4 running in Docker inside the container, with its socket directory bind-mounted so PHP on the host connects over a local Unix socket rather than TCP.Redis for caching and sessions.Our existing reverse proxy in front, terminating TLS with a Let's Encrypt certificate.Putting MySQL in Docker was deliberate. We wanted a specific MySQL version that doesn't depend on the distribution's packaging, and upgrades that are a container image tag instead of a package migration. A script instead of a checklistThe whole build is one shell script. It creates the container, installs packages, configures nginx and PHP-FPM, starts MySQL with its data on a dedicated volume, creates the database and user, installs Redis, and sets PHP limits for uploads and memory. The only manual part is running Invision's web installer and entering the licence. If the container dies, we can rebuild it and restore the data in well under an hour. Writing the script took longer than doing the install by hand would have, but only the first time. Making it look like oursWe themed it to the Harbor Tech Bytes logo colours and set up separate light and dark logos, because a logo that looks right on white often disappears on a dark background. The palette is defined once in the theme settings, so headers, buttons and links stay consistent in both modes. BackupsA nightly job dumps the database with a consistent single-transaction dump, archives the uploads directory and configuration, and ships both off the container. On top of that, Proxmox backs up the whole container every night, so we can restore either a single database or the entire thing. We've tested the restore, because I don't count a backup as real until I've restored from it. Where the install fought back An installer error that wasn't about the installerThe web installer failed at the database step with an error that looked like a problem with the installer itself. It turned out PHP couldn't find the MySQL socket. PHP was looking at the default socket path, and with MySQL in a container the socket only shows up on the host if the directory is bind-mounted and the permissions let PHP-FPM's user reach it. Once we shared the socket directory properly and set the path explicitly in the configuration, the database step passed. Then the installer stalled at a session check, where it verifies that PHP sessions persist between requests. That one was on the PHP side too. Sessions weren't being saved anywhere the next request could read them. Once PHP had a working, writable session store, re-running the installer got us through. Both failures were obvious in the PHP-FPM and nginx error logs and misleading on screen. An installer tells you which step failed, not why, so check the logs before you believe the message. A custom page that threw a server errorWe built a custom page for the site and it returned a server error instead of content. Friendly (clean) URLs weren't enabled in our install, so nothing could resolve the page's pretty URL. Turning on friendly URLs in the admin panel, along with the matching nginx rewrite that routes requests to Invision's front controller, fixed it. The rest of the site's URLs got nicer as a bonus. Behind the proxyLike every app behind a reverse proxy, the forum first saw every visitor as coming from the proxy's address. Configuring nginx to trust the proxy's forwarded headers gave us real client IPs back, which matters for spam protection and moderation. So now we have a forum that's fast, searchable and ours, running in a single container next to the rest of our infrastructure. There's a rebuild script and a tested restore, branding that holds up in light and dark mode, and you're reading the result. If you're running something like this yourself, my main advice is to check the PHP and nginx logs before you start debugging the installer.

One of our municipal clients had a remote-access VPN appliance that, as far as anyone could remember, had never worked properly. Staff in the VPN group would connect and get a portal page with a single remote-desktop link. If they launched the full VPN client instead, it connected and then dropped within about a second. Over the years people had just learned to use the remote-desktop link and live without mapped drives, internal web apps and anything else that needed a real tunnel. When we asked about it, the answer was basically "oh, the VPN's always been like that." On paper the design was fine. Domain laptops with a valid machine certificate get a full network tunnel, and everything else gets remote desktop only. We needed to work out why every laptop was being treated as "everything else". Following one person's sessionWe didn't start with overall connection stats. We followed a single user's session from start to finish in the appliance's per-session logs. The access policy ran its steps in order. It authenticated against AD, checked the client's machine certificate, ran an endpoint antivirus check, and then assigned resources. The certificate check failed on every session, on every laptop, every time. The policy then sent the user down the fallback branch that only granted the remote-desktop link, and the client tunnel got torn down because no network-access resource had been assigned. The aggregate numbers just said connections were succeeding. Only the session log showed which step failed and which branch the user ended up on. The laptops were fineThe obvious theory was that the laptops didn't have certificates. They did. Every domain laptop had a valid computer certificate, auto-enrolled from the client's internal AD Certificate Services root, current, and with client-authentication usage. I'm glad we checked first, because otherwise we could easily have spent days rebuilding certificate templates that were already working. An empty trust profileThe certificate check on the appliance pointed at a certificate-authority profile, which tells it which CA certificates to trust when validating a client certificate. That profile existed and was attached to the policy, but it had no CA certificate loaded at all, so it could never validate anything. Years of "the VPN is flaky" came down to that one empty field. What we changedWe loaded the internal root CA into the trust profile used by the certificate check.We moved the group check to the front of the policy. While mapping the policy we found a dormant branch. Once the certificate check started passing, a valid machine certificate would have led to a full tunnel without checking VPN group membership on that path. If we'd only fixed the CA, anyone with a domain laptop would have had full access. With the AD group check first, users outside the group never reach the certificate step.We softened the antivirus check. A failed AV check used to deny access entirely, which locked people out when their definitions were a day behind. Now it falls back to remote-desktop-only, so the person can still work and the problem shows up in the logs.We fixed the public certificate chain. The appliance's public TLS certificate had been installed without its intermediate. Most browsers coped by fetching it themselves, but some clients didn't, and that explained a separate batch of intermittent connection errors. We installed the full chain.We documented it all, including the policy flow as a diagram, which CA the check trusts, where the certificates come from, and when each one expires.Things that slowed us down On many appliances of this type, saving a policy edit doesn't make it live until you explicitly apply the policy. We now apply, then confirm the change actually took effect in the next session log instead of trusting the editor.Users with existing sessions kept the old behaviour until their session was killed. We cleared the test users' sessions between every test.The fix depends on auto-enrolment staying healthy. A laptop that sits off the network long enough for its certificate to expire drops back to remote-desktop-only. That's the correct behaviour, but it's something to watch now, so we added certificate expiry on long-offline devices to our regular checks.Where it stands nowDomain laptops in the VPN group get a full tunnel with mapped drives and internal apps, for the first time since the appliance went in. Users outside the group are denied correctly and consistently, instead of by accident because of a broken check. The access hole that would have opened the moment someone "fixed" the certificate check got closed before it ever went live. The help desk also stopped getting a whole category of tickets. The part I'd pass on is about fixing a check that has always failed. Before you turn it on, read every branch it feeds. A check that always fails hides whatever is sitting on its success path, and nobody has looked at that path in years.

Like a lot of small MSPs, our hosting grew one server at a time. We had a VPS for the ticketing system, another for the monitoring stack, and one for the RMM because it wanted its own box. The SIEM got another because it wanted a lot of RAM, a couple more ran WordPress sites and small internal apps, and the documentation wiki and password manager got squeezed in wherever there was room. Each decision made sense at the time. Together they came to about $334 a month across two providers, with backups, patching and firewall rules done slightly differently on every box. Whenever something broke, the first question was always "which server is that one on again?" We also had retired hyperconverged hardware sitting around, a four-node chassis that came out of a customer refresh and was perfectly healthy. So the plan was to move everything onto hardware we own, in a colo rack, while our clients' tickets, alerts and logins kept working the whole time. The platformWe wiped the four nodes and installed Proxmox VE with Ceph across all of them, so any VM or container can run on any node and survive a node failure. The vendor's own hypervisor stack went away. Proxmox gave us clustering, live migration and built-in backups, with no licensing to keep track of. How the network is laid outA VyOS VM is the router and firewall. It holds a public /28 that the colo routes to us.Services live on an internal VLAN behind it. Management interfaces sit on a separate VLAN that isn't reachable from the internet at all.Nginx Proxy Manager takes ports 80 and 443 on one public IP and terminates TLS for every HTTP service.The few services that really need their own address get a 1:1 NAT from a dedicated public IP on the /28. That's mainly the RMM, which has its own agent ports, and the SIEM, for agent enrolment and event ports.One service per containerEach service got its own LXC container, or a full VM where the software insisted (mainly the RMM and the SIEM). Monitoring (LibreNMS and Grafana), the wiki, the password manager, the ticketing system and each small app all became separate containers. Snapshots, restores and resource limits are now per-service, and a bad PHP upgrade in one app can't take another one down with it. Moving things without anyone noticingWe had one rule for every migration. Prove the new origin works before you touch DNS. Build the new container and restore data from the old server.Test it by pointing a hosts-file entry, or a curl with an explicit resolve, at the new address. That way you're exercising the real hostname and certificate.Do a final data sync during a short write freeze where needed (the ticketing and password manager databases).Flip DNS. We'd set low TTLs a day ahead, so most clients moved within minutes.Leave the old server running, read-only where possible, until traffic to it stops.Nothing was deleted on cutover day. We shut the old servers down and kept provider snapshots for rollback, and only destroyed each one once its service had run cleanly on the new cluster for a while. Those snapshots cost a few dollars, and it was very nice knowing we could roll back. What tripped us up Apps that didn't trust the proxy. Several applications, LibreNMS included, generated wrong redirect URLs or treated every visitor as coming from the proxy's IP until we told them which proxies to trust. Each app has its own setting for this, and checking for it is now part of every migration.Old addresses were hard-coded all over the place. Webhooks, alert transports and integrations still pointed at the old servers' IPs or hostnames. Monitoring alerts went quiet for a while because a webhook target was still the old box, which is not a great feeling for an MSP. Now we grep every config and database for old IPs and hostnames before decommissioning anything.The RMM expected a certificate covering several subdomains, and a per-host certificate caused agent connection errors. A wildcard certificate fixed it.Adding an interface to the VyOS VM changed the order the guest saw its NICs, which swapped our inside and outside networks on reboot. We pin interfaces to MAC addresses in the router config now.Moving lots of data over SSH tripped fail2ban on the old servers, so we were locking ourselves out. Batching commands into fewer connections and whitelisting our office IP sorted it.Where it landedMonthly hosting went from about $334 to about $112, and most of what's left is leftover services we're still winding down. The goal is close to zero beyond colo and transit. Every service is now backed up, monitored and patched the same way, on the same schedule. Restores are faster too. A container snapshot rollback takes seconds, and backups are local instead of being pulled across the internet. Clients didn't see any outages during the move. If you're planning something similar, start with the inventory. Our first list of what ran where was missing about a third of the cron jobs, webhooks and little scripts living on those servers. And go in knowing that owning the hardware means owning its problems. We got the cost savings and the control, and we also took on power, spare parts and hardware monitoring. I'd rather plan for a node dying now than figure it out the night it happens.

Bulk renames look trivial. Loop over the files, work out a new name, move. The trouble starts when two inputs map to the same output, because a plain mv or Move-Item -Force overwrites the first file with the second and says nothing. We learned this the hard way normalising a folder of scanned documents for a client. Invoice 001.pdf and invoice_001.pdf both became invoice_001.pdf, and one of them was simply gone. These days every batch job I write follows the same habits. Here they are, with working bash and PowerShell versions. Before you run either script bash 4 or newer for the associative array in the bash version. macOS ships bash 3.2, so install a newer bash or drop the collision map.PowerShell 5.1 or 7.x for the PowerShell version.A copy of the data, or at least a backup, for the first real run.The rules I follow nowDry run by default. The script only prints what it would do unless you explicitly tell it to act.Detect collisions in the plan, before anything moves.Never clobber. Check the destination, and use mv -n as a backstop.Log every change with old and new names, so you can undo it.Make it re-runnable. Files already in the target state get skipped, so running it twice is harmless and an interrupted run can just be restarted.Bash#!/usr/bin/env bash # normalize-names.sh - lowercase names and replace spaces with underscores # Usage: ./normalize-names.sh DIR (dry run) # ./normalize-names.sh DIR --apply (actually rename) set -euo pipefail DIR="${1:?usage: $0 DIR [--apply]}" APPLY=0 [[ "${2:-}" == "--apply" ]] && APPLY=1 LOG="rename-$(date +%Y%m%d-%H%M%S).log" declare -A planned # target -> source, needs bash 4+ while IFS= read -r -d '' src; do dir=$(dirname "$src"); base=$(basename "$src") new=$(printf '%s' "$base" | tr '[:upper:]' '[:lower:]' | tr ' ' '_') [[ "$base" == "$new" ]] && continue # already done: re-runnable dest="$dir/$new" if [[ -n "${planned[$dest]:-}" ]]; then echo "COLLISION $src and ${planned[$dest]} -> $dest" | tee -a "$LOG" continue fi planned[$dest]="$src" if [[ -e "$dest" ]]; then echo "EXISTS $src -> $dest (skipped)" | tee -a "$LOG" continue fi if (( APPLY )); then mv -n -- "$src" "$dest" if [[ -e "$src" ]]; then echo "FAILED $src -> $dest" | tee -a "$LOG" else echo "RENAMED $src -> $dest" | tee -a "$LOG" fi else echo "WOULD $src -> $dest" | tee -a "$LOG" fi done < <(find "$DIR" -maxdepth 1 -type f -print0)A few details in there are easy to miss: find -print0 with read -d '' handles spaces, quotes and even newlines in names. Never loop over $(ls).mv -n (no clobber) refuses to overwrite an existing file. I don't trust its exit status to tell me it skipped, because that behaviour differs between platforms and coreutils versions, so the script checks whether the source still exists afterwards.The -- before the filenames stops a file called -rf.txt being read as options.PowerShellPowerShell gives you dry runs for free. Declare SupportsShouldProcess, wrap the change in $PSCmdlet.ShouldProcess(), and -WhatIf works just like it does on built-in cmdlets. # Normalize-Names.ps1 - run with -WhatIf first [CmdletBinding(SupportsShouldProcess)] param([Parameter(Mandatory)][string]$Path) $log = "rename-{0:yyyyMMdd-HHmmss}.csv" -f (Get-Date) $plan = Get-ChildItem -LiteralPath $Path -File | ForEach-Object { $new = $_.Name.ToLower() -replace '\s+', '_' [pscustomobject]@{ Source = $_.FullName; NewName = $new; Target = Join-Path $_.DirectoryName $new } } | Where-Object { (Split-Path $_.Source -Leaf) -cne $_.NewName } # skip already-normalized # Two inputs mapping to the same output? Stop before touching anything. $collisions = $plan | Group-Object Target | Where-Object Count -gt 1 if ($collisions) { foreach ($c in $collisions) { Write-Warning "Collision -> $($c.Name): $($c.Group.Source -join ', ')" } throw 'Resolve collisions before renaming.' } $results = foreach ($item in $plan) { if (Test-Path -LiteralPath $item.Target) { [pscustomobject]@{ Source = $item.Source; Target = $item.Target; Result = 'SkippedExists' } continue } if ($PSCmdlet.ShouldProcess($item.Source, "Rename to $($item.NewName)")) { try { Rename-Item -LiteralPath $item.Source -NewName $item.NewName -ErrorAction Stop $r = 'Renamed' } catch { $r = "Failed: $($_.Exception.Message)" } [pscustomobject]@{ Source = $item.Source; Target = $item.Target; Result = $r } } } if ($results) { $results | Export-Csv -LiteralPath $log -NoTypeInformation -WhatIf:$false }Run it in two passes: .\Normalize-Names.ps1 -Path D:\Scans\Incoming -WhatIf .\Normalize-Names.ps1 -Path D:\Scans\IncomingRename-Item already refuses to overwrite an existing file, but we still test first so the skip lands in the log instead of showing up as an error. Keep Move-Item -Force out of rename scripts. -Force is exactly the overwrite you're trying to prevent. Read the plan before you apply itWhen the dry run finishes, I look for any COLLISION or warning lines first. Then any EXISTS lines, since a target that's already there usually means an earlier partial run or a genuine duplicate. Last, I check the count against what I expected. If the plan says it'll rename 4,000 files and you expected 40, stop. Did anything go missing? Count files before and after. The numbers must match. Renames never reduce the file count unless something got overwritten.Run the script again in dry-run mode. It should report nothing to do.Keep the log with the data. To undo, swap the columns and feed it back through the same no-clobber logic.Where this has caught us out Case-only renames on case-insensitive filesystems. On NTFS and default macOS APFS, Report.PDF to report.pdf looks like the destination "already exists" because it's the same file, so both scripts above will skip it. If case-only changes matter, go through a temporary name (Report.PDF to Report.PDF.tmp to report.pdf).The in-script map only catches collisions within one run. The destination-exists check is what covers files left by earlier runs.Write logs somewhere other than the target folder, or the next run may try to rename its own log.OneDrive, Dropbox and DFS replication can lag or create conflict copies during a big rename. Pause sync or run it on the server side.I use the same pattern for other batch jobs too, like bulk AD changes, mailbox moves and clearing out old backups. Show the plan, check for collisions, then act, log it, and make sure a second run won't hurt.

Every AD we inherit has the same ghosts. There are computer accounts for laptops recycled years ago, users who left before the last audit, and "temp" accounts that quietly became permanent. A stale-object report is one of the first things we run in a new environment, and after that we schedule it monthly. Here's the script, plus the caveats that keep you from disabling the PC the front desk still logs into every morning. What you need A domain-joined machine with the RSAT Active Directory module (Import-Module ActiveDirectory).A normal domain user account. It can read these attributes, so you don't need Domain Admin just to report.A share or folder to save the reports to.Why lastLogonTimestamp lies a littleAD has two attributes that matter here. lastLogon is precise but not replicated. Each domain controller keeps its own value, so you'd have to query every DC and take the newest.lastLogonTimestamp is replicated, so one query gives a usable answer, but it's deliberately imprecise. It only updates when the existing value is older than ms-DS-Logon-Time-Sync-Interval (14 days by default) minus a random 0 to 5 days. In practice it can lag the real last logon by roughly 9 to 14 days.For a "not seen in 90 days" report that doesn't matter at all. For "did this user log in yesterday?" it's useless. Pick thresholds well above 14 days. Turning the number into a dateThe attribute is a Windows FILETIME (100-nanosecond intervals since 1 January 1601), and [DateTime]::FromFileTime() converts it to local time. Accounts that have never logged on have no value, and converting 0 gives you a date in 1601, so handle that explicitly. The AD module also exposes a ready-converted LastLogonDate property, which is fine to use. I convert manually so the behaviour is obvious to whoever reads the script after me. The script# Get-StaleADReport.ps1 Import-Module ActiveDirectory $Days = 90 $Cutoff = (Get-Date).AddDays(-$Days) $Stamp = Get-Date -Format 'yyyy-MM-dd' $OutDir = '\\fileserver.contoso.com\IT\Reports\AD' function Convert-LLT($value) { if ($value -and $value -gt 0) { [DateTime]::FromFileTime($value) } else { $null } } $computers = Get-ADComputer -Filter * -Properties LastLogonTimestamp, OperatingSystem, Enabled, Description | ForEach-Object { [pscustomobject]@{ Name = $_.Name Enabled = $_.Enabled OS = $_.OperatingSystem LastLogon = Convert-LLT $_.LastLogonTimestamp Description = $_.Description DN = $_.DistinguishedName } } | Where-Object { -not $_.LastLogon -or $_.LastLogon -lt $Cutoff } | Sort-Object LastLogon $users = Get-ADUser -Filter 'Enabled -eq $true' -Properties LastLogonTimestamp, whenCreated | ForEach-Object { [pscustomobject]@{ SamAccountName = $_.SamAccountName Name = $_.Name LastLogon = Convert-LLT $_.LastLogonTimestamp Created = $_.whenCreated DN = $_.DistinguishedName } } | Where-Object { (-not $_.LastLogon -and $_.Created -lt $Cutoff) -or ($_.LastLogon -and $_.LastLogon -lt $Cutoff) } | Sort-Object LastLogon $computers | Export-Csv "$OutDir\stale-computers-$Stamp.csv" -NoTypeInformation $users | Export-Csv "$OutDir\stale-users-$Stamp.csv" -NoTypeInformation "Stale (> $Days days): $($computers.Count) computers, $($users.Count) enabled users"Look at the user filter. An account that has never logged on but was created yesterday isn't stale, so for those we compare whenCreated instead. Getting the report to someoneWe save dated CSVs to a restricted share so we can compare month to month. If you want it emailed, know that Microsoft has marked Send-MailMessage obsolete and it doesn't handle modern authentication well. For Microsoft 365 tenants we send through Graph's sendMail using an app registration with Mail.Send, scoped with an application access policy so it can only send as the reporting mailbox. A relay you control works too. Either way, attach the CSV or link to the share. Nobody wants hundreds of rows pasted into an email body. Sanity checks before anyone disables anything Pick three machines from the report and look them up in your RMM or endpoint manager. If they've been online recently, something's off. Check whether they're really using a different computer account name.Pick one machine you know is used every day and confirm it's not on the report.Cross-check a handful of users against HR's leaver list.Then go in stages. Disable the object, move it to a "Pending Delete" OU with a note in the description, wait 30 days, and only then delete. Disabling is reversible. Deleting without the Recycle Bin is not. A matching bug that made a report look cleanThis one isn't about lastLogonTimestamp, but it came out of the same kind of cleanup. We were comparing a device export against a client list to find machines assigned to the wrong client. Using -like "*$client*" meant a client called "Riverton" also matched "North Riverton", so devices were silently counted under the wrong client and the report looked fine. Match names exactly, with -eq or a lookup set: # BAD: substring / wildcard match - "Riverton" also matches "North Riverton" $devices | Where-Object { $_.ClientName -like "*$client*" } $devices | Where-Object { $_.ClientName -match $client } # regex, same problem # GOOD: exact, case-insensitive comparison $devices | Where-Object { $_.ClientName -eq $client } # GOOD for large lists: a HashSet with a case-insensitive comparer $known = [System.Collections.Generic.HashSet[string]]::new( [string[]]$clientList.Name, [System.StringComparer]::OrdinalIgnoreCase) $unmatched = $devices | Where-Object { -not $known.Contains($_.ClientName.Trim()) }Accounts that look stale but aren't Service accounts and appliances often authenticate in ways that don't update lastLogonTimestamp the way you'd expect, or they log on rarely. Exclude a known service-account OU or review those by hand.Cluster and virtual objects (cluster name objects, some SQL listener accounts) can look stale while being critical. Read the description and DN before you disable one.Computer password age is a useful second opinion. Domain members change their machine password every 30 days by default, so a PasswordLastSet more than a few months old on a computer account is a strong sign the machine is gone.Hybrid-joined devices may also show up in Entra ID. Clean up both, or the stale device keeps reappearing in your Intune reports.

Running a script on one machine is easy. Running it on four hundred machines across a dozen clients, where half of them are laptops that drift in and out of the office, is how you find out whether the script was any good. We use Tactical RMM (TRMM) for this, but most of what follows carries over to any RMM. It's mostly about knowing what account the script runs as, getting output you can actually read afterwards, and not pushing to everyone on the first try. What to have ready A working Tactical RMM instance with agents deployed.An account that can manage scripts and run them on agents.An API key (Settings > Global Settings > API Keys) if you want to drive it from your own tooling.One or two test machines you're allowed to break.Keep a library, not a pile of one-offsAnything you run more than once goes in the Script Manager with a clear name, a category, a description of what it changes, and a sensible default timeout. TRMM supports PowerShell, batch and Python. Ad-hoc commands from the console are fine for poking around, but if it changes state it belongs in the library, where someone else can review it and re-run it exactly the same way. Remember it runs as SYSTEMBy default TRMM scripts run as SYSTEM. That's what you want for installing software, changing services, editing HKLM or anything else machine-wide. It also means a few things catch people out: HKCU is SYSTEM's hive, not the user's.Mapped drives and user-profile paths aren't what you expect.Network access goes out as the computer account.The Run As User option runs the script in the logged-on user's session instead. It only works when somebody is actually signed in, so it's a poor fit for an overnight fleet run. For per-user settings we usually run as SYSTEM and loop through the user hives, or deploy the change somewhere that applies at sign-in. One script with parameters beats five copiesInstead of keeping five near-identical scripts, accept parameters and pass them from the script's default arguments or at run time: param( [string]$ServiceName = 'Spooler', [ValidateSet('Check','Restart')][string]$Mode = 'Check' )Arguments are typed the way you would on a command line (for example -ServiceName Spooler -Mode Restart). Never pass secrets as plain arguments, because they end up in run history. Use the RMM's custom fields or environment variables where you can, and limit who can view history. Write it so running it twice is harmlessA fleet script will hit machines in every state you can imagine, and someone will run it twice. Check before you change anything, make a second run a no-op, and report clearly. We start every output line with a fixed status word so results can be grouped later, and we always set an exit code: # Ensure-BitLockerRecoveryEscrow.ps1 - safe to run repeatedly $ErrorActionPreference = 'Stop' try { $vol = Get-BitLockerVolume -MountPoint $env:SystemDrive if ($vol.ProtectionStatus -ne 'On') { Write-Output "SKIP: BitLocker not enabled on $env:SystemDrive" exit 0 } $kp = $vol.KeyProtector | Where-Object KeyProtectorType -eq 'RecoveryPassword' if (-not $kp) { Write-Output "FAIL: no recovery password protector" exit 1 } BackupToAAD-BitLockerKeyProtector -MountPoint $env:SystemDrive -KeyProtectorId $kp[0].KeyProtectorId | Out-Null Write-Output "OK: escrowed $($kp[0].KeyProtectorId)" exit 0 } catch { Write-Output "FAIL: $($_.Exception.Message)" exit 1 }Don't skip the exit code. When a script is used as a check or an automated task, the RMM uses the return code to decide pass or fail. A script that catches every error and exits 0 will show green forever, and whoever's watching the dashboard has no idea machines are failing underneath it. Roll it out in ringsOrder matters here. This is the sequence we stick to: Run on one test machine and read the full output.Run on a small, mixed group: a desktop, a laptop, an older OS build, a machine with a non-admin user.Use bulk script against a single client or site.Only then target everything, or attach it to an automation policy.Put recurring work on a policyFor anything recurring, add the script as an automated task on an automation policy instead of running it by hand every week. Policies apply at the client, site or agent level, so you can pilot on one site before widening it. Offline agents pick up their tasks when they come back, which beats chasing laptops around manually. Counting the results Export the bulk results and count statuses. Eyeballing hundreds of rows is how things get missed. # results.csv exported from the bulk run: hostname, output Import-Csv .\results.csv | ForEach-Object { $status = ($_.output -split ':')[0].Trim() [pscustomobject]@{ Host = $_.hostname; Status = $status; Detail = $_.output } } | Group-Object Status | Select-Object Name, CountEvery machine should land in OK, SKIP or FAIL. Anything with no output at all usually timed out or was offline, so rerun those separately. Calling TRMM from your own scriptsOnce you're comfortable, you can trigger runs from your own tooling. Authentication is an X-API-KEY header. Check field names against your version, since they've changed over time. The easiest way I've found is to watch the request the web UI sends in your browser's developer tools. $base = 'https://api.rmm.contoso.com' $headers = @{ 'X-API-KEY' = (Get-Secret -Name 'TrmmApiKey' -AsPlainText) } # Cloudflare Browser Integrity Check blocked our POSTs without this $ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36' # GET worked even without the User-Agent $agents = Invoke-RestMethod -Uri "$base/agents/" -Headers $headers -UserAgent $ua # POST returned 403 / error 1010 until -UserAgent was added $body = @{ script = 42 # script ID from the library args = @('-Mode', 'Check') output = 'wait' timeout = 120 run_as_user = $false } | ConvertTo-Json $agentId = '<AgentID>' Invoke-RestMethod -Method Post -Uri "$base/agents/$agentId/runscript/" ` -Headers $headers -UserAgent $ua -ContentType 'application/json' -Body $bodyThe 1010 error, and a few others Our RMM API sits behind Cloudflare. GET requests worked perfectly, which made the next part confusing. Every POST and DELETE came back 403 with Cloudflare error 1010. Error 1010 means Cloudflare blocked the request based on the client's browser signature, usually via Browser Integrity Check. Scripted clients send a non-browser User-Agent, and only the write requests were being judged that harshly. Sending a normal browser User-Agent header fixed it straight away. The cleaner long-term fix is a WAF rule that skips Browser Integrity Check for the API hostname, restricted to your own source IPs. Timeouts are often too short. Installers on slow laptops easily run past a 30-second default, and a timed-out script may still be running on the endpoint."It works when I run it" usually means it works as you, in your session, with your drives mapped. SYSTEM is a different world.Keep output short and structured. A script that dumps a full event log for every machine makes the results unreadable.

Most of our tenant reporting runs overnight: device inventories, licence counts, stale-account checks. Nobody is sitting there to answer an MFA prompt at 2 a.m., and a job tied to one tech's sign-in breaks the week that person gets a new phone or moves on. So we run these jobs as app registrations using the OAuth2 client credentials flow. The script signs in as the app itself, gets a token, and calls Graph with plain Invoke-RestMethod. No modules are needed, which I appreciate on locked-down jump boxes where installing anything is its own ticket. This is the small script we start from. It lists users and Intune managed devices, follows paging properly, and keeps the secret out of the file. Before you start An account that can create app registrations and grant admin consent (Application Administrator, plus a Privileged Role Administrator or Global Administrator to consent to Graph application permissions).PowerShell 5.1 or 7.x.The Microsoft.PowerShell.SecretManagement and Microsoft.PowerShell.SecretStore modules if you go with a client secret.Registering the appIn the Entra admin center, go to Applications > App registrations > New registration.Give it a name the next tech will understand (we use something like svc-graph-reporting). Single tenant, no redirect URI.Copy the Application (client) ID and Directory (tenant) ID from the Overview page.Permissions, and the consent button everyone forgetsUnder API permissions > Add a permission > Microsoft Graph, pick Application permissions, not Delegated. Delegated permissions are for a signed-in user and do nothing in the client credentials flow. This script needs two: User.Read.AllDeviceManagementManagedDevices.Read.AllThen click Grant admin consent for <tenant>. This is the step I always forget on a new tenant. Until the status column shows a green tick, you'll still get a token, but every call comes back 403. Ask only for the read permissions you actually need. An app secret with Directory.ReadWrite.All is a tenant-wide key sitting on some server, and I don't want to be the one explaining that to a client. Where the credential livesUnder Certificates & secrets you can upload a certificate or create a client secret. I prefer certificates, because the private key stays in the machine's certificate store and never gets pasted into a file. If you do use a secret, give it a short expiry, put the renewal date on a calendar someone actually reads, and never hard-code it. We keep ours in a SecretStore vault on the box that runs the job: Install-Module Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore -Scope CurrentUser Register-SecretVault -Name LocalStore -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault Set-Secret -Name 'GraphReportApp' -Secret (Read-Host -AsSecureString 'Client secret')Run that as the same account the scheduled task runs as, because the vault is per-user. For unattended use you'll also need to set the SecretStore's password behaviour to suit your environment (see Set-SecretStoreConfiguration). Getting a token and following the pagesThe token endpoint is https://login.microsoftonline.com/<TenantID>/oauth2/v2.0/token. With client credentials the scope is always https://graph.microsoft.com/.default, which means "whatever application permissions have been consented". The response contains access_token, which is good for roughly an hour. Graph hands results back in pages. When there are more, the response includes an @odata.nextLink property with the full URL of the next page, and you keep following it until it's gone. If you skip this, the report quietly stops at the first page (100 users by default). Nobody notices for months, until someone asks why the licence count looks low. The full script# Get-GraphInventory.ps1 - list users and Intune managed devices via Microsoft Graph # Requires: Microsoft.PowerShell.SecretManagement + a registered vault holding the secret $TenantId = '<TenantID>' $ClientId = '<AppClientID>' $Secret = Get-Secret -Name 'GraphReportApp' -AsPlainText function Get-GraphToken { $body = @{ client_id = $ClientId scope = 'https://graph.microsoft.com/.default' client_secret = $Secret grant_type = 'client_credentials' } $tokenUri = "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" (Invoke-RestMethod -Method Post -Uri $tokenUri -Body $body ` -ContentType 'application/x-www-form-urlencoded').access_token } function Invoke-GraphPaged { param([Parameter(Mandatory)][string]$Uri, [Parameter(Mandatory)][hashtable]$Headers) $all = [System.Collections.Generic.List[object]]::new() do { $page = Invoke-RestMethod -Method Get -Uri $Uri -Headers $Headers foreach ($item in $page.value) { $all.Add($item) } $Uri = $page.'@odata.nextLink' } while ($Uri) $all } $headers = @{ Authorization = "Bearer $(Get-GraphToken)" } # Single quotes matter: in double quotes PowerShell would try to expand $select $users = Invoke-GraphPaged -Headers $headers ` -Uri 'https://graph.microsoft.com/v1.0/users?$select=displayName,userPrincipalName,accountEnabled&$top=999' $devices = Invoke-GraphPaged -Headers $headers ` -Uri 'https://graph.microsoft.com/v1.0/deviceManagement/managedDevices?$select=deviceName,operatingSystem,osVersion,lastSyncDateTime,userPrincipalName' "{0} users, {1} managed devices" -f $users.Count, $devices.Count $devices | Sort-Object lastSyncDateTime | Select-Object deviceName, operatingSystem, osVersion, lastSyncDateTime, userPrincipalName | Export-Csv -Path .\managed-devices.csv -NoTypeInformationIf you'd rather use a certificateDoing certificate auth with raw Invoke-RestMethod means building and signing a JWT client assertion yourself. It's doable but fiddly, so when we use certificates we let the Microsoft Graph PowerShell SDK do that part: # Certificate auth with the Graph SDK - no secret on disk at all Connect-MgGraph -TenantId '<TenantID>' -ClientId '<AppClientID>' -CertificateThumbprint '<Thumbprint>' Get-MgDeviceManagementManagedDevice -All | Select-Object DeviceName, OperatingSystem, LastSyncDateTimeIt's the same app registration with the same permissions. You just upload the public key (.cer) under Certificates & secrets. Checking the token before you trust the report $t = Get-GraphToken $t.Length # a long string, not an error ($t -split '\.')[1] # middle segment of the JWT - decode it to check the "roles" claimDecode the middle part of the token with a JWT decoder you trust, or base64-decode it locally. The roles claim should list User.Read.All and DeviceManagementManagedDevices.Read.All. If roles is missing, consent hasn't been granted.Compare the device count in the CSV with the count in the Intune admin center. If yours comes out at exactly 100, 999 or 1000, paging is broken.Things that have bitten us Double-quoted URLs. "...users?$select=..." expands $select to an empty string, and Graph happily returns every property. Use single quotes, or escape the dollar sign with a backtick.Long runs outlive the token. A big tenant can take more than an hour to page through, so on long jobs request a fresh token before the old one expires instead of reusing one for the whole run.Throttling. Push Graph too hard and it returns 429 with a Retry-After header. Wrap your calls in a retry that honours that header rather than hammering it.Expired secrets. This is the most common "the report stopped" ticket we get. The error is AADSTS7000222 (expired client secret). Track expiry dates somewhere you actually look.Adding the Delegated version of a permission looks right in the portal and does nothing for this flow.

Certificates expire on a schedule you know years ahead, and expired certificates are still one of the most common outages we get called about. My favorite example was a public VPN portal whose certificate had expired roughly six months before anyone told us. Users had just been clicking through the warning. Nothing was monitoring it, because it was an appliance, not a web server anyone thought of as a website. When we replaced the certificate, a second problem showed up. The appliance wasn't serving the intermediate certificate. Browsers papered over that, but stricter clients refused to connect. Here's how we check for both problems by hand and with a small script. Gather this first A Linux or macOS shell with opensslA list of every TLS endpoint you're responsible for, including websites, VPN portals, mail servers, firewall and appliance admin pages, and RDP gatewaysSomewhere to run a scheduled job and send its output, like email, a chat webhook or your monitoring systemSee what the server is really sendingConnect and show the certificates the server sends. Always pass -servername so SNI picks the right certificate on servers that host more than one name. Without it, a shared server may hand you its default certificate, and the check looks fine while the real site is broken. openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com -showcerts </dev/nullThe output lists each certificate the server sent, numbered from 0 (the leaf). Each one shows s: (subject) and i: (issuer). A healthy chain from a public CA usually has the leaf at 0 and one or more intermediates after it. If only certificate 0 appears and its issuer is an intermediate CA, the server isn't sending its chain. Reading the expiry datePipe the leaf into openssl x509 to read its dates: echo | openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com 2>/dev/null \ | openssl x509 -noout -subject -issuer -enddatesubject=CN = vpn.contoso.com issuer=C = US, O = Example CA, CN = Example Intermediate R1 notAfter=Mar 10 23:59:59 2027 GMTIn scripts, I lean on -checkend SECONDS. It exits non-zero if the certificate expires within that many seconds, so you don't have to parse dates or deal with the differences between GNU and BSD date. Checking the chainLook at the verification result at the end of the s_client output: openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com </dev/null 2>/dev/null | grep 'Verify return code'You want Verify return code: 0 (ok). The failures we see most: 21 (unable to verify the first certificate) or 20 (unable to get local issuer certificate) usually means the server is missing an intermediate.10 (certificate has expired) means what it says.18 or 19 means a self-signed certificate, or a private CA this machine doesn't trust. For internal hosts, pass -CAfile with your root.So why don't browsers complain about a missing intermediate? Many browsers cache intermediates or fetch them using the Authority Information Access URL in the leaf. Command-line tools, VPN clients, Java and Python applications, mail servers and older mobile devices often don't, so they fail while the browser looks fine. That's why I never test with a browser alone. The fix is to install the full chain on the server, meaning the leaf plus intermediates but not the root. The script we run every morningPut your endpoints in a text file, one host port per line: vpn.contoso.com 443 mail.contoso.com 443 portal.contoso.com 8443 fw01.contoso.com 443Then run this script from cron daily: #!/usr/bin/env bash # cert-check.sh - warn on certificates expiring soon or with broken chains # usage: cert-check.sh hosts.txt [warn_days] LIST="${1:-hosts.txt}" WARN_DAYS="${2:-30}" SECS=$((WARN_DAYS * 86400)) status=0 while read -r host port; do [[ -z "$host" || "$host" == \#* ]] && continue port="${port:-443}" out=$(timeout 15 openssl s_client -connect "$host:$port" -servername "$host" </dev/null 2>/dev/null) cert=$(printf '%s\n' "$out" | openssl x509 2>/dev/null) if [[ -z "$cert" ]]; then echo "ERROR $host:$port no certificate retrieved"; status=2; continue fi end=$(printf '%s\n' "$cert" | openssl x509 -noout -enddate | cut -d= -f2) verify=$(printf '%s\n' "$out" | grep -m1 'Verify return code' | sed 's/^ *//') if ! printf '%s\n' "$cert" | openssl x509 -noout -checkend 0 >/dev/null; then echo "EXPIRED $host:$port $end"; status=2 elif ! printf '%s\n' "$cert" | openssl x509 -noout -checkend "$SECS" >/dev/null; then echo "WARN $host:$port expires $end (under $WARN_DAYS days)"; status=1 else echo "OK $host:$port expires $end" fi if [[ "$verify" != *"0 (ok)"* ]]; then echo "CHAIN $host:$port $verify"; [[ $status -lt 1 ]] && status=1 fi done < "$LIST" exit $statusSchedule it so it only emails when the exit code is non-zero. That way, a certificate email in your inbox always means something needs attention. 0 7 * * * /opt/scripts/cert-check.sh /opt/scripts/hosts.txt 30 > /tmp/cert-check.out || mail -s 'Certificate check' noc@contoso.com < /tmp/cert-check.outMost monitoring systems can do this natively too, with HTTP checks on certificate age or a TLS sensor. If you already have one, use it. We still find the script handy for appliances on odd ports and as a second opinion. Make sure the check itself works Run the script against a host you know is fine and confirm it prints OK with the right date.Point it at a test site with a known-bad certificate (badssl.com has expired and incomplete-chain examples) and confirm you get EXPIRED and CHAIN lines and a non-zero exit code.Confirm the cron email actually arrives.Where certs slip past us Renewal isn't installation. Automated renewal on one box doesn't update the copy someone manually loaded onto a load balancer or appliance.Missing intermediates hide from browsers. Trust Verify return code over what the browser shows you.For us the hard part wasn't the script, it was building the list. VPN gateways, firewall admin pages, UPS cards and printers are the ones that expire unnoticed, so build the list from DNS and your firewall NAT rules rather than from memory.

Most of our small clients will never pay for an expensive SIEM, but they still deserve someone watching their logs. Wazuh is how we cover that. It's free and open source, and it gives a small MSP a lot of what the big products offer, across Windows, Linux and macOS. We run a single Wazuh manager for every client. This post covers how we deploy agents from the RMM, keep clients separated, cut down the noise, and keep the manager itself from turning into a liability. What we started with A Wazuh server (manager, indexer and dashboard). For a few hundred agents, the all-in-one install on a VM with 8 vCPU and 16 GB RAM or more is a good start.Agents that can reach the manager on TCP 1514 (events) and 1515 (enrollment)An RMM or script deployment tool for Windows and Linux endpointsWhat you get out of itLog collection and analysis. Agents ship Windows event logs, syslog and application logs to the manager, which decodes them and matches them against thousands of rules.File integrity monitoring (FIM), which alerts when watched files, directories or registry keys changeVulnerability detection. Agents inventory installed packages and the manager compares them against CVE feeds.Security configuration assessment (SCA), which scores endpoints against CIS-style benchmark checksActive response, which is optional automated action such as blocking an IP after repeated failures. I'd be careful with this one.Lock down enrollment firstOut of the box, anything that can reach port 1515 can enroll as an agent. Turn on password enrollment on the manager. In /var/ossec/etc/ossec.conf, inside the <auth> section, set <use_password>yes</use_password>, then write the password to the auth file: echo 'REPLACE-WITH-LONG-RANDOM-STRING' > /var/ossec/etc/authd.pass chmod 640 /var/ossec/etc/authd.pass chown root:wazuh /var/ossec/etc/authd.pass systemctl restart wazuh-managerOne group per clientGroups let you push different configuration to different agents, and they let whoever is looking at the dashboard filter down to one client. Create them on the manager before you deploy anything: /var/ossec/bin/agent_groups -a -g client-contoso -q /var/ossec/bin/agent_groups -a -g client-fabrikam -q /var/ossec/bin/agent_groups -lEach group gets a shared config file at /var/ossec/etc/shared/GROUP/agent.conf, and agents in that group pull it automatically. That's where client-specific FIM paths or extra log sources go. Pushing agents from the RMMThe agent packages accept deployment variables, so one command installs the agent, enrolls it and assigns the group. Match the agent version to your manager version or older, because an agent newer than the manager won't connect. On Windows: $msi = "$env:TEMP\wazuh-agent.msi" Invoke-WebRequest -Uri 'https://packages.wazuh.com/4.x/windows/wazuh-agent-4.x.x-1.msi' -OutFile $msi Start-Process msiexec.exe -Wait -ArgumentList "/i `"$msi`" /q WAZUH_MANAGER=`"wazuh.contoso.com`" WAZUH_REGISTRATION_PASSWORD=`"REPLACE-ME`" WAZUH_AGENT_GROUP=`"client-contoso`"" Start-Service -Name WazuhSvcReplace 4.x.x with the exact version you run. On Debian or Ubuntu, with the Wazuh repository already added: WAZUH_MANAGER='wazuh.contoso.com' WAZUH_REGISTRATION_PASSWORD='REPLACE-ME' \ WAZUH_AGENT_GROUP='client-contoso' apt-get install -y wazuh-agent systemctl daemon-reload systemctl enable --now wazuh-agentKeep the registration password in an RMM secret or script variable instead of hard-coding it in a script every technician can see. The first week is loudExpect a flood at first. Don't disable whole rule groups. Write targeted overrides in /var/ossec/etc/rules/local_rules.xml instead, where a child rule with level 0 suppresses a specific match. For example, this silences successful logons from a known service account: <group name="local,windows,"> <rule id="100100" level="0"> <if_sid>60106</if_sid> <field name="win.eventdata.targetUserName">^svc-backup$</field> <description>Ignore routine logons by the backup service account</description> </rule> </group>Use custom rule IDs from 100000 upward. Test by pasting a sample event into /var/ossec/bin/wazuh-logtest, then restart the manager. Treat the manager like a domain controllerYour SIEM holds logs from every client and can push configuration to every agent. If someone gets into it, they get into everyone. Don't expose the dashboard to the internet without SSO and MFA in front of it, or keep it reachable only over VPN.Change the default admin passwords for the dashboard, indexer and API right after install.If agents connect over the internet, expose only 1514 and 1515, and consider restricting them to client office IPs.Keep the API port (55000) internal.Back up /var/ossec/etc (keys, rules, groups), and plan indexer retention so the disk doesn't fill.Checking agents landed in the right place /var/ossec/bin/agent_control -l /var/ossec/bin/agent_groups -l -g client-contosoAgents should show as Active and appear in the right group. In the dashboard, filter by group and confirm events, SCA results and vulnerability data are coming in. Then create a test file in a monitored directory and watch for the FIM alert. Things that caught us out An agent newer than the manager won't connect. Upgrade the manager first, every time.Cloned VMs can carry duplicate agent keys. A template that already has an enrolled agent causes name and ID collisions, so remove the agent's client.keys before sealing a template.The indexer disk fills up. Set index lifecycle policies early.Vulnerability alerts pile up fast. We report vulnerabilities per client instead of alerting on each CVE.Each week I look at the top rules by count in the dashboard and tune whichever one is worst. It only takes a few minutes, and it's what keeps our techs reading the alerts instead of muting them.

LibreNMS is what we use to watch switches, routers, firewalls, UPSes and access points across our client sites. When a closet switch drops, we want the alert to reach whoever is on call before the office staff come in and find it. LibreNMS auto-discovers a lot over SNMP, graphs ports and health sensors out of the box, and can alert to chat, ticketing and paging systems. This is how we set up SNMPv3 on a device, add it, write alert rules and wire up transports. I've also included a few transport bugs that cost us alerts before we caught them. What to have ready A working LibreNMS install. We run it in a container, but a native install is the same from here on.Network reachability from the LibreNMS poller to the device on UDP 161Admin access to the deviceWebhook or API details for wherever alerts should go, such as Discord, your ticket system or PagerDutySNMPv3 on the deviceUse SNMPv3 with authentication and privacy (authPriv). SNMPv2c community strings travel in clear text. Here's a generic Cisco IOS example with a read-only view and an ACL limiting who can poll: ip access-list standard SNMP-POLLERS permit 10.10.20.5 ! snmp-server view LIBRENMS-VIEW iso included snmp-server group LIBRENMS-GRP v3 priv read LIBRENMS-VIEW access SNMP-POLLERS snmp-server user librenms LIBRENMS-GRP v3 auth sha AUTH-PASSPHRASE priv aes 128 PRIV-PASSPHRASE snmp-server location Main Office MDF snmp-server contact noc@contoso.comOn IOS, SNMPv3 users don't show up in the running config, so put the credentials in your password manager right away. You can check the user with show snmp user. It's also worth setting a meaningful location string, because LibreNMS uses it for maps and grouping. Prove SNMP works before LibreNMS gets involvedI always test with snmpwalk from the LibreNMS host before adding a device. It tells you whether you have an SNMP problem or a LibreNMS problem: snmpwalk -v3 -l authPriv -u librenms -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1 sysDescrIf that returns the device description, add it from the web UI (Devices, Add Device) or the CLI: cd /opt/librenms sudo -u librenms ./lnms device:add --v3 -l authPriv -u librenms \ -a SHA -A 'AUTH-PASSPHRASE' -x AES -X 'PRIV-PASSPHRASE' 10.10.20.1Discovery runs, and the device fills in ports, sensors and hardware details over the next few polling cycles. A short list of alert rulesLibreNMS ships a set of default rules you can import from Alerts, Alert Rules. Start small. An alert stream nobody reads is worse than none. These are the rules we always deploy: Device down, meaning device status is 0 with ICMP and SNMP both failingPort down on uplinks, scoped to ports with a description you tag rather than every access portSensor over limit, for temperature, PSU and UPS battery stateDevice rebooted, meaning uptime below a few minutes, which catches power eventsUse device groups, for example one per client or site, to scope rules and route them to the right transport. Set a sensible delay so a single missed poll doesn't page anyone. Where the alerts goTransports are what deliver alerts. Under Alerts, Alert Transports, add one per destination: For Discord, paste the channel webhook URL. It's good for team visibility.Most ticket systems can create tickets by email, or through an API transport that POSTs JSON to an endpoint.For PagerDuty, use the integration key from a PagerDuty service. We only page for real outages, so nobody gets woken up for a warning.Map rules to transports directly on the rule or through transport groups. Press the Test button on each transport before you rely on it. Proving it end to end Run the validator and fix anything it flags: cd /opt/librenms sudo -u librenms ./validate.phpThe device shows recent poll times and the graphs are filling in.Each transport's Test button delivers a message.Unplug a lab device or shut a test port, and confirm the alert fires, appears in the alert log, and clears when you put it back.The bugs that cost us alerts On the version we were running, an API transport saved without its options field populated (even as empty) threw a PHP error when an alert fired. It didn't just skip that transport, it stopped the whole alert run. If alerts stop arriving everywhere at once, go looking for one broken transport.A PagerDuty transport created without the region field set caused the same kind of error and killed the run. Open each PagerDuty transport, set the region explicitly, and save it.If you write your own webhook receiver, look at the actual JSON. Fields like alert state arrived as strings in our payloads, and a receiver comparing them to integers silently dropped every alert.There's no automatic retry. If a transport endpoint is down when the alert fires, that notification may simply be lost, so monitor your receivers too.The alert log in the UI shows what fired. When something fired but never arrived, the answer is usually in /opt/librenms/logs/librenms.log.If you add a second poller, add it to the device ACL too. Otherwise it fails with timeouts that look a lot like wrong credentials.After any change to transports, I press Test on every one of them. One broken transport can silence all the others, and you won't find out until a real outage goes unreported.

This started as a routine look at a client's remote access portal logs. We found around 50,000 password-spray sessions a day coming from just five networks, with zero successful logins. Nobody had noticed, because spraying is quiet by design. Instead of hammering one account with thousands of passwords, the attacker tries a few common passwords against thousands of usernames, which stays under per-account lockout thresholds. Anything with a public login page gets this as constant background noise. Here's how we spotted it, measured it and shut it off at the edge. Access you'll want The portal or VPN authentication logs, whether that's local logs, a syslog server or a SIEMA Linux shell with the usual tools (grep, awk, sort, uniq)Admin access to the edge firewall or load balancer in front of the portalWhat it looks like in the logsOnce you've seen it, the pattern is easy to recognize: Thousands to tens of thousands of authentication sessions per day, far more than your user count could explainLots of usernames, often ones that don't exist in your directory, like admin, test, scanner, or first.last guesses from a scraped staff listA handful of source /24 networks, often hosting providers, rotating through addresses within each blockNo successes at all. If any attempts succeed, stop reading and start incident response.Counting failures by networkEvery product logs differently, so you'll need to adapt the field extraction. Here's a generic log line: 2026-10-01T03:12:44Z portal auth=fail user=jsmith src=203.0.113.45 reason=bad_passwordCount failed attempts per source /24: grep 'auth=fail' portal.log \ | grep -oE 'src=[0-9.]+' | cut -d= -f2 \ | awk -F. '{print $1"."$2"."$3".0/24"}' \ | sort | uniq -c | sort -rn | head -20Then check how many distinct usernames each network tried. A real user behind a NAT fails with one or two names. A sprayer tries hundreds. grep 'auth=fail' portal.log \ | awk '{for(i=1;i<=NF;i++){if($i~/^user=/)u=substr($i,6); if($i~/^src=/)s=substr($i,5)} split(s,o,"."); print o[1]"."o[2]"."o[3]".0/24", u}' \ | sort -u | awk '{print $1}' | uniq -c | sort -rn | head -20And confirm there were no successes from those networks: grep 'auth=success' portal.log | grep -E 'src=(203\.0\.113|198\.51\.100)\.' | wc -lFor this client, five networks accounted for essentially all of the failures, each had tried hundreds of usernames, and the success count was zero. Block it before it reaches login and MFAWhere you block matters. If the portal forwards each attempt to your MFA provider or directory, the spray costs you more than log space. It loads your domain controllers, can lock out real staff whose usernames happen to match, and can spam people with push prompts if a password ever does match. Drop the traffic before it gets to the login logic. Most platforms have a named list you can block against. It might be an address list or address group on a firewall, a data group referenced by a load balancer rule, or an external dynamic list. Keep the list in one place so adding a network is a one-line change. On a Linux edge, ipset does the same job: ipset create spray-block hash:net ipset add spray-block 203.0.113.0/24 ipset add spray-block 198.51.100.0/24 iptables -I INPUT -p tcp --dport 443 -m set --match-set spray-block src -j DROPMake the rule persistent using your distribution's mechanism, and put a comment or ticket reference on each entry. Whoever looks at that list next will want to know why it's there. Geo-blocking, if your users are all in one countryBlocking five networks handles today's sprayer, and the next one will come from somewhere else. If every legitimate user is in one country, allowing only that country to reach the portal removes most of the noise with one rule. Most commercial firewalls have built-in geo objects. Keep an exception process for staff who travel. Geo-IP data is approximate and VPN providers can appear to be anywhere, so I treat this as noise reduction, not a security control on its own. MFA and lockout still do the real workEdge blocking cuts load and noise, but it doesn't protect accounts. Attackers rotate sources, and spraying is slow on purpose. What actually protects your users: MFA on every remote access path, preferably phishing-resistant or number-matching, so a guessed password alone isn't enoughA lockout or smart-lockout policy that slows guessing without making it trivial to lock out the whole staffBanned-password lists so seasonal passwords like Fall2026! get rejectedAlerts on any successful login from an unusual country or network, and on MFA prompts users report they didn't triggerNext-day checks Re-run the per-network count on the next day's log. The blocked networks should be gone.The firewall rule's hit counter should be climbing.Real users can still log in, including from home ISPs and mobile carriers.The MFA provider logs no longer show a stream of authentication requests for unknown users.Be careful with these Before blocking a network, check that it isn't a mobile carrier or a large ISP your users share. Hosting provider ranges are usually safe to block. Residential ranges are not.If a load balancer sits in front of the portal, make sure the real client IP is being logged. Otherwise every attempt looks like it came from one internal address.A block rule placed below a broad allow rule never matches. Check the hit counts after you add it.The list is easy to forget about. Review the blocked networks every few months and remove stale entries.We re-run the per-network count on that portal every so often. New sprayers keep turning up from new places, and adding another /24 to the list takes about a minute.

When we inherit a Proxmox cluster, backups are one of the first things I look at. Proxmox makes them easy to turn on, which also makes it easy to assume you're covered when you aren't. This is what we run for our own cluster and for clients: which vzdump mode we pick, how we schedule and prune, why I think Proxmox Backup Server is worth the extra box, and the restore testing that tells you whether any of it works. Have these ready A Proxmox VE node or clusterBackup storage that is not on the same disks or pool your VMs run on. An NFS share or a separate disk works, and a Proxmox Backup Server (PBS) is better.A spare VM ID range for restore tests. We use 9000 and up.A snapshot isn't a backupA Proxmox snapshot is a point-in-time state stored on the same storage as the VM. It's perfect for "I'm about to upgrade this and want to roll back in five minutes if it goes badly." It does nothing for you if the storage fails, the pool fills, someone deletes the VM, or ransomware reaches the host. Long-lived snapshots also slow down some storage types and eat space as the disk changes. Take them before a change and delete them after. Backups are separate copies on separate storage. Picking a vzdump modeSnapshot mode keeps the guest running while it's backed up. For VMs, Proxmox uses QEMU's live backup mechanism, and with the QEMU guest agent installed and enabled it briefly freezes the guest filesystems for consistency. For containers it needs storage that supports snapshots, or it falls back to suspend mode. This is our default.Suspend mode, for containers, copies data while running and then suspends the container briefly for a final sync. For VMs it offers little over snapshot mode and causes a pause.Stop mode shuts the guest down, backs it up and starts it again. It's the most consistent, at the cost of downtime. We use it for a few legacy systems whose databases don't tolerate live backup well.Install and enable the QEMU guest agent on every VM: qemu-guest-agent on Linux, the VirtIO guest tools on Windows, and the Guest Agent option in the VM settings. Without it, snapshot-mode backups are crash-consistent, not filesystem-consistent. A one-off manual backup looks like this: vzdump 101 --mode snapshot --storage backup-nfs --compress zstd --notes-template '{{guestname}} manual'Scheduling, and why hand-picked lists go wrongCreate jobs under Datacenter, Backup in the GUI. They're stored cluster-wide in /etc/pve/jobs.cfg and run on whichever node holds each guest. We usually run one nightly job for everything, outside business hours, selecting by pool or by "all" with exclusions instead of a hand-picked list of VM IDs. Hand-picked lists are how a new VM ends up never backed up. Point the job's failure notifications at someone who actually reads that inbox. A backup job that's been failing silently for three weeks is the most common problem we find on inherited clusters. How long to keep thingsRetention is set with prune options on the job or on the storage. For most small environments we start here: keep-last=3, keep-daily=7, keep-weekly=4, keep-monthly=6On the command line the same thing is: vzdump 101 --storage backup-nfs --prune-backups keep-daily=7,keep-weekly=4,keep-monthly=6That gives you a week of dailies, a month of weeklies and half a year of monthlies. The monthlies are what save you when someone notices that data went missing back in the spring. Proxmox Backup Server is worth the extra boxBacking up to a plain NFS share works, but every backup is a full compressed archive. PBS changes that in a few ways: Data is stored in chunks and identical chunks are stored once, so nightly backups of mostly unchanged VMs take a small fraction of the space.With dirty bitmaps, a running VM only sends changed blocks after the first backup, so jobs finish much faster.Verify jobs re-read chunks and check their checksums on a schedule, which catches silent corruption before you need the backup.Sync jobs to a second PBS offsite give you the offsite copy in a 3-2-1 setup.You also get file-level restore from the GUI, encryption, and per-user permissions.On PBS, schedule prune, garbage collection and verify jobs per datastore. Prune only marks old snapshots for removal. Garbage collection is what actually frees the space. Restore to a new ID and actually boot itA backup you've never restored is a hope, not a backup. Restore to a new VM ID so the production guest is left alone: # VM from a vzdump archive on file storage qmrestore /mnt/pve/backup-nfs/dump/vzdump-qemu-101-2026_10_01-02_00_03.vma.zst 9101 --storage local-lvm --unique # VM from PBS (volume ID as shown by pvesm list) pvesm list pbs01 --vmid 101 qmrestore pbs01:backup/vm/101/2026-10-01T09:00:03Z 9101 --storage local-lvm --unique # Container pct restore 9201 /mnt/pve/backup-nfs/dump/vzdump-lxc-201-2026_10_01-02_10_44.tar.zst --storage local-lvmBefore you boot the copy, take it off the network. Otherwise it can fight the production guest for its IP or start sending email. qm set 9101 --net0 virtio,bridge=vmbr0,link_down=1 qm start 9101When you rewrite net0, keep the NIC model and bridge that qm config 9101 shows. Then open the console and confirm the OS boots, services start and the data is current. Delete the test guest when you're done. What a healthy setup looks like The backup job's task log shows every expected guest with status OK.The storage view lists backups at the retention points you expect.On PBS, the last verify job is green.You have a dated note of the last successful restore test for each important system.Mistakes we keep running into Backups sitting on the same pool as the VMs. If Ceph or the RAID dies, both go together. Backup storage has to be independent.Restoring over the original by accident. Restoring to the existing VM ID overwrites it, so always use a new ID unless you mean to replace production.Duplicate MAC and IP. Without --unique and a disconnected NIC, a restored VM can knock the original off the network.Trusting snapshot mode for databases. A filesystem freeze isn't the same as an application-consistent database backup, so run native dumps for critical databases as well.A full PBS datastore. If garbage collection isn't running, pruned backups still take up space until the datastore fills.We restore something every month. If the dated note for an important system is more than a month old, that restore is the next thing on my list.

Over a few years we'd piled up a fleet of cloud VPS instances: a monitoring box, a ticketing box, a couple of web hosts, a mail relay, and a scattering of one-off servers. Each was cheap on its own, but together they cost us about $334 a month. We consolidated everything onto our own Proxmox cluster in a colocation rack, and the hosting bill dropped to about $112 a month. That's roughly two thirds less, and it keeps heading toward near zero as the last stragglers move. Honestly, the bigger win for us was having one place to back up, monitor and patch. Here's the method that let us do it without the people we support seeing any downtime. What we had in place first A Proxmox cluster with enough capacity and a backup targetRoutable public addresses at the new location. We use a small public block.A router or firewall VM in front of the cluster. We use VyOS, and OPNsense or pfSense work too.Control of DNS for every hostname being movedA spreadsheet. Seriously.Write down what every box really runsBefore moving anything, log into every VPS and write down what it actually runs, not what you remember it running. For each box we captured: Listening ports and the processes behind them (ss -tlnup)Every hostname pointing at it, from DNS and from web server configsCron jobs and systemd timers for every userDatabases, where their data lives, and how big they areInbound dependencies like webhooks, firewall allowlists at third parties, and API clients that connect by IPOutbound dependencies like SMTP relays and API keys bound to source IPsThe cron jobs and IP allowlists are what get you. A forgotten nightly report, or a vendor that only accepts connections from your old IP, won't show up until a week after cutover when someone asks where their report went. One container per serviceI get the urge to rebuild the old all-in-one boxes as they were. Resist it. We gave each service its own LXC container (or a VM where it needed one) with its own IP on an internal services VLAN, for example 10.10.20.0/24. Now each service gets backed up, restored, upgraded and moved on its own, and the next migration becomes moving one small thing at a time. You own the firewall nowYour VPS provider was giving you a firewall for free, even if you never thought about it. Now it's your job. We run a VyOS VM with the public uplink on one side and the internal VLANs on the other. The policy is simple. Inbound is dropped by default, established and related return traffic is allowed, and only the specific ports each published service needs are opened. Internal VLANs can reach the internet, but the internet reaches nothing unless a rule says so. A single reverse proxy for the web stuffMost of what we moved was HTTP. Instead of giving every web app a public IP, we run one reverse proxy container and forward ports 80 and 443 to it. Ours is Nginx Proxy Manager, though Caddy, Traefik or plain Nginx are just as good. It terminates TLS, holds the certificates, and routes by hostname to the right internal container, so one public IP can serve dozens of sites. 1:1 NAT for the few that need their own IPSome things don't fit behind a shared HTTP proxy. Think mail servers that need matching reverse DNS, VPN endpoints, game or voice servers on fixed ports, and anything a third party allowlists by IP. For those we use 1:1 NAT on the router, mapping a dedicated public address to one internal container, with firewall rules limited to the ports that service uses. Keep this list short, because every 1:1 mapping is one more exposed surface to watch. Test the new side while DNS still points at the old oneThis is the step that made our cutovers boring, which is exactly what you want. Copy the data, bring the service up on the new side, and test it while DNS still points at the old server. Lower the DNS TTL to 300 seconds a day or two ahead so the eventual flip propagates quickly. From inside the network, hit the backend directly with a Host header: curl -sI -H 'Host: app.contoso.com' http://10.10.20.15/Then test the full public path, including the router, reverse proxy and TLS certificate, without changing DNS. --resolve pins the hostname to the new public IP: curl -sv --resolve app.contoso.com:443:192.0.2.10 https://app.contoso.com/ -o /dev/null 2>&1 | grep -E 'HTTP/|subject:|expire'You can do the same thing in a browser by temporarily adding the name to your hosts file. Log in, click around, submit a form, and check that background jobs run. Only once it works end to end do we touch DNS. Flip DNS, but keep a way backFor services with a database, do a final sync in a quiet window. Stop the app on the old side, take a last dump, restore it on the new side, then flip DNS. After that: Stop the old VPS. Don't delete it. A stopped instance can be started again in a minute if something was missed.Take a provider snapshot of the old box as a second rollback and keep it until you're confident. Snapshots usually cost a fraction of a running instance.If you can, watch the old box's logs for a few days before stopping it. Any traffic still arriving points you at a hard-coded IP or a forgotten client.Signs the move actually stuck dig +short app.contoso.com returns the new address from several resolvers.The certificate on the new proxy is valid and auto-renewing.Cron jobs, scheduled reports and outbound email are firing from the new location.Each new container shows up in your backup jobs and monitoring.Things that surprised us Several integrations called our old server by IP rather than hostname. The access logs on the box we kept running a little longer are how we found them.Apps behind the proxy suddenly see every request coming from the proxy's IP, which breaks rate limiting and fail2ban. Configure trusted proxies and X-Forwarded-For handling in each app.A new public IP has no sending history and probably no matching PTR record. Fix reverse DNS and SPF before moving anything that sends mail.Don't cancel the VPS account first. Cancel the instances, keep the snapshots, and close the account only once you no longer need them.Moving one small service at a time felt slow while we were doing it. Looking back, it's the reason nobody had to call us about it.