Further down is a related trick I use from the RMM a lot: getting a machine, and scripts running as SYSTEM on it, to pick up a new group membership without rebooting someone's PC in the middle of their day.

Delegating resets and unlocks

You'll need Domain Admin once to set it up, users organised into OUs so you can scope it, and a security group for the people getting the rights, for example GRP-Helpdesk-PwReset. Always delegate to a group, never to individuals. When someone joins or leaves the help desk you change group membership, not ACLs.

1. In Active Directory Users and Computers, right-click the OU containing the users (for example OU=Staff,DC=contoso,DC=local) and choose Delegate Control.
2. Add GRP-Helpdesk-PwReset.
3. Under common tasks, tick Reset user passwords and force password change at next logon.
4. Finish the wizard.

That common task doesn't include unlocking, which catches people out. Run the wizard again on the same OU, choose Create a custom task to delegate, then Only the following objects in the folder, tick User objects, and on the permissions page show Property-specific permissions and tick Read lockoutTime and Write lockoutTime.

If you're setting this up across a lot of OUs or clients, I'd script it with dsacls instead:

$ou  = "OU=Staff,DC=contoso,DC=local"
$grp = "CONTOSO\GRP-Helpdesk-PwReset"

dsacls $ou /I:S /G "${grp}:CA;Reset Password;user"
dsacls $ou /I:S /G "${grp}:RPWP;pwdLastSet;user"
dsacls $ou /I:S /G "${grp}:RPWP;lockoutTime;user"

/I:S applies the entry to child objects only, CA is a control access right (Reset Password), and RPWP is read and write property.

The help desk can then use ADUC from RSAT, or PowerShell:

Set-ADAccountPassword -Identity jdoe -Reset -NewPassword (Read-Host -AsSecureString "New password")
Set-ADUser -Identity jdoe -ChangePasswordAtLogon $true
Unlock-ADAccount -Identity jdoe

New group membership for SYSTEM, no reboot

Computer accounts get group memberships too, usually to grant access to a file share, a certificate template or a GPO's security filter. The catch is that a computer gets its Kerberos tickets when it boots and renews them for hours afterwards. Add it to a group and it won't "see" the membership until it gets a new ticket, which normally means a reboot. That's a hard sell when someone is halfway through their work on that machine.

You can force it by purging the tickets for the SYSTEM logon session (LUID 0x3e7). The next time anything running as SYSTEM needs a ticket, the machine asks the DC for a fresh TGT, and that one carries the new group.

First add the computer to the group. Note the trailing $ on the computer's account name.

Add-ADGroupMember -Identity "GRP-Deploy-Share-Read" -Members "WS-0142$"

Make sure the change has replicated to the DC the machine authenticates against (repadmin /syncall /APed if you're in a hurry). Then on the target machine, from an elevated prompt or your RMM, which usually runs as SYSTEM anyway:

klist -li 0x3e7 purge
gpupdate /force          # if the group is used for GPO security filtering
certutil -pulse          # if the group grants enrollment on a certificate template

Now the script running as SYSTEM can reach the share or service that's restricted to the group. Access the resource by its DNS name, not its IP address, so Kerberos actually gets used.

If the membership was only for a one-off job, like copying installers from a restricted share or a single enrollment, take the computer back out when it's done and purge again so the access goes away now and not in ten hours:

Remove-ADGroupMember -Identity "GRP-Deploy-Share-Read" -Members "WS-0142$" -Confirm:$false
# then on WS-0142:
klist -li 0x3e7 purge

Put the removal in the same script or ticket as the add. We've found computers sitting in "temporary" groups months after the job, because nobody came back for them.

I mostly use the purge for remote one-off jobs where a reboot would interrupt someone, for testing GPO filtering or certificate autoenrollment without rebooting each time, and for temporary access during a deployment script. It isn't a substitute for proper standing access. If machines need a group permanently, put them in it and let them pick it up on their next reboot like normal.

It's messy in quieter ways too. The logs show that person "logging in" thousands of times a day, and because those accounts are often admins, the appliance ends up holding admin credentials. What we do instead is boring on purpose. One dedicated service account per system, minimum rights, a long random password, no interactive logon, and a written rotation process.

To do this you'll need rights to create OUs, users and GPOs, a password manager or vault for the generated password, and access to the config of whatever system is going to bind.

Give service accounts their own OU

Keeping them out of your normal user OUs means user GPOs, password policies and cleanup scripts don't touch them by accident.

New-ADOrganizationalUnit -Name "Service Accounts" -Path "OU=Admin,DC=contoso,DC=local" -ProtectedFromAccidentalDeletion $true

Create the account

Generate the password in your password manager first. We go 32 characters or more. Name the account so whoever finds it in two years knows what it's for, and put the owner and runbook in the description.

$pw = Read-Host "Password for svc-vpn-ldap" -AsSecureString

New-ADUser -Name "svc-vpn-ldap" `
    -SamAccountName "svc-vpn-ldap" `
    -UserPrincipalName "svc-vpn-ldap@contoso.local" `
    -Path "OU=Service Accounts,OU=Admin,DC=contoso,DC=local" `
    -AccountPassword $pw `
    -Enabled $true `
    -PasswordNeverExpires $true `
    -CannotChangePassword $true `
    -Description "LDAP bind for remote access portal. Owner: IT. See runbook SVC-001."

"Password never expires" is a deliberate trade-off. An expiring password on a service account causes exactly the outage we're trying to avoid. The compensating controls are the long random password and the rotation process further down.

Keep it to Domain Users

For read-only lookups (authenticate a user, read their group memberships), plain Domain Users membership is enough, since authenticated users can read most user and group attributes by default. Please don't add it to Domain Admins or any admin group "to make it work". If an app genuinely needs to write, like a password self-service tool, delegate that specific permission on the specific OU and nothing else.

Block interactive logon

A bind account should never be able to sit at a desktop or open an RDP session. Make a group, something like GRP-Deny-Interactive-Logon, and add the account. Then in a GPO linked at the domain level (or the OUs holding your servers and workstations), go to Computer Configuration, Policies, Windows Settings, Security Settings, Local Policies, User Rights Assignment and add the group to Deny log on locally and Deny log on through Remote Desktop Services.

Don't add it to Deny access to this computer from the network on your DCs. LDAP binds are network logons, so that breaks the one thing the account exists to do.

Use TLS, and test before you touch the appliance

Set the appliance to LDAPS (636) or StartTLS if it supports it. A simple bind over plain 389 sends the password in clear text. Before I change anything on the device, I test the bind from any Linux box with the OpenLDAP client tools:

ldapsearch -H ldaps://dc01.contoso.local -D "svc-vpn-ldap@contoso.local" -W \
  -b "DC=contoso,DC=local" "(sAMAccountName=jdoe)" memberOf

If that returns the user's groups, the account works. Then update the appliance, test a real user's login, and only then remove the old credentials.

Before you disable the old shared account, watch the DC logs for its logon attempts for a few days. There's always a second device nobody remembered, and in my experience it's usually a scan-to-email printer.

Write it down so rotation isn't scary

For each service account we record what uses it and where the credential is entered (device, admin URL, config file), who owns the integration, where the password is stored, and when it was last rotated along with the steps. Rotation then is just generating a new password, updating the vault, setting it in AD, updating each consumer and testing. One account per system keeps that list short, and one rotation can't break five things.

People sometimes ask why not use a gMSA. They're great for Windows services, but the password is retrieved by Windows hosts, so a VPN appliance or printer can't use one. For appliances, a normal user account managed like this is the right tool.

The design is simple once you see it. The DC holding the PDC emulator role in the forest root domain syncs to a reliable external NTP source. Every other DC syncs from the PDC emulator, or another DC higher up the hierarchy. Member servers and workstations sync from a DC in their domain, and that part happens on its own for domain-joined machines. So nearly all the work is on the PDC. You'll need admin access to your DCs, outbound UDP 123 from the PDC emulator to the NTP servers you pick, and about fifteen minutes.

Find the PDC and see what it's doing

netdom query pdc
# or
(Get-ADDomain).PDCEmulator

Then on the PDC emulator:

w32tm /query /status
w32tm /query /source
w32tm /query /configuration

Read the Source and Stratum lines. If Source says Free-running System Clock, the server has no time source at all and just trusts its own clock. Local CMOS Clock is the same problem, it's reading the hardware clock. VM IC Time Synchronization Provider means it's getting time from the Hyper-V host, which I'll come back to.

Point the PDC at external NTP

w32tm /config /manualpeerlist:"time.nist.gov,0x8 0.pool.ntp.org,0x8 1.pool.ntp.org,0x8" /syncfromflags:manual /reliable:yes /update
net stop w32time
net start w32time
w32tm /resync /rediscover

/manualpeerlist is a space-separated list of peers. The ,0x8 suffix tells Windows to use client mode with them, which is what public NTP servers expect. /syncfromflags:manual says to use that list instead of the domain hierarchy. /reliable:yes advertises this DC as a reliable time source to the rest of the domain, and /update tells the service to load the new config.

I always list more than one peer, so a single unreachable server doesn't leave you free-running again.

If the status still shows the local clock afterwards, check that UDP 123 is actually getting out. w32tm /stripchart /computer:time.nist.gov timing out is a good sign the firewall is blocking it. And if the PDC is already far off, Windows may refuse to jump the clock because of the maximum correction limits (MaxPosPhaseCorrection and MaxNegPhaseCorrection). Set the time by hand close to correct, then resync.

Put everyone else back on the hierarchy

Run this on every other DC, and on any member server someone "fixed" by hand at some point:

w32tm /config /syncfromflags:domhier /update
net stop w32time
net start w32time
w32tm /resync
w32tm /query /source

The source should now be a DC name, not an internet server or the local clock.

Virtual DCs shouldn't trust the host

A virtual DC shouldn't take its time from the hypervisor. The host's clock is often the thing that's wrong, and then the whole domain is following an unmanaged box nobody ever set up NTP on. On Hyper-V, the cleanest fix that keeps the integration service's other benefits is to disable only the VM IC time provider inside the guest:

reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t REG_DWORD /d 0 /f
net stop w32time
net start w32time
w32tm /query /source

You can also untick Time synchronization under Integration Services in the VM settings. On VMware, make sure periodic time sync with the host is off in VMware Tools for DC guests, and give the ESXi hosts proper NTP so any one-off syncs (after a vMotion or snapshot revert, for example) are at least correct.

w32tm /query /status
w32tm /monitor
w32tm /stripchart /computer:time.nist.gov /samples:5 /dataonly

If you've got ten seconds today, run w32tm /query /source on your PDC. I'm still surprised how often the answer is wrong.

You'll want a DC or a management box with RSAT (AD DS tools), an account in Domain Admins (or delegated rights to read replication metadata), and an elevated prompt or PowerShell window.

Start with dcdiag across the whole enterprise

dcdiag runs a battery of tests against a DC: connectivity, replication, advertising, FSMO reachability, SYSVOL and more. I always run it verbose and save it to a file, because the lines that matter scroll past way too fast on screen.

dcdiag /v /c /e > C:\Temp\dcdiag-all.txt
dcdiag /test:dns /v /e > C:\Temp\dcdiag-dns.txt

/e tests every DC in the enterprise and /c runs the comprehensive set. Don't skip /e. Without it you're only testing the box you're logged into. We had one domain where the local DC was spotless and a branch office DC hadn't replicated in 40 days.

Search the file for failed test. Some failures are noise. SystemLog, for example, fails if there's been any error in the System log in the last hour. Anything in Replications, Advertising, NetLogons, KccEvent or DFSREvent deserves a proper look.

Then the replication summary

repadmin /replsummary

You get one table showing, per DC, the largest replication delta and how many of its partners are failing. You want zero in the fails column and deltas in minutes, not days. If any delta is getting close to your tombstone lifetime (180 days on modern forests), stop and fix that first. A DC that's been out of replication longer than that must not just be reconnected.

Per-partner detail when something looks off

repadmin /showrepl
repadmin /showrepl * /csv > C:\Temp\showrepl.csv
repadmin /queue

/showrepl lists each naming context (Domain, Configuration, Schema, DomainDnsZones, ForestDnsZones) and each inbound partner with the result of its last attempt. Once you have more than a handful of DCs, the CSV is much easier to filter. /queue shows pending inbound requests. If the queue never drains, that DC is overloaded or can't reach its partner.

The errors I see most here are 1722 (RPC server unavailable, usually firewall or DNS), 8453 (access denied on replication) and 8524 (DSA operation unable to proceed because of a DNS lookup failure). I check DNS first every time. Most replication errors turn out to be DNS errors in disguise.

Forcing replication

repadmin /syncall /APed

The flags are A for all partitions, P to push changes outward from this DC, e to include partners in other sites, and d to identify servers by distinguished name in the output. Run it on the DC where the change was made. Every partition should report "SyncAll terminated with no errors". Then rerun repadmin /replsummary to confirm.

FSMO roles

netdom query fsmo

You should get five roles, all held by DCs that actually exist and are online. If one points at a DC that was removed without being demoted, you'll need to seize that role to a healthy DC with ntdsutil and clean up the dead DC's metadata. While you're here, note which DC is the PDC emulator. It matters for time sync and password changes.

Speaking of dead DCs, leftover DNS records and connection objects from DCs that were simply switched off throw replication errors that look alarming but are really a metadata cleanup job. Remove the dead DC in AD Sites and Services and clean up its DNS records instead of chasing each error.

SYSVOL and DFSR

dfsrmig /getglobalstate
net share
Get-CimInstance -Namespace root\microsoftdfs -ClassName DfsrReplicatedFolderInfo |
    Select-Object ReplicatedFolderName, State

dfsrmig /getglobalstate should say Eliminated, which means the migration from FRS to DFSR is complete. Anything else means you're still on FRS, or halfway off it, and newer Windows Server versions won't accept that as a DC. net share has to show both SYSVOL and NETLOGON. A DC missing those won't process Group Policy for clients, and that's usually how users find out for you. In the CIM query, a State of 4 means Normal, 2 is initial sync and 5 is in error.

Also check the DFS Replication event log on each DC for event 2213, which means replication paused after a dirty shutdown. It needs a manual resume, and it's the one I'm most likely to forget.

I run this monthly and keep the output files. Comparing this month's against last month's is how you catch the slow drift before it turns into a Monday morning outage.