Catching expired certs and missing intermediates before your users do

Certificates expire on a schedule you know years ahead, and expired certificates are still one of the most common outages we get called about. My favorite example was a public VPN portal whose certificate had expired roughly six months before anyone told us. Users had just been clicking through the warning. Nothing was monitoring it, because it was an appliance, not a web server anyone thought of as a website. When we replaced the certificate, a second problem showed up. The appliance wasn't serving the intermediate certificate. Browsers papered over that, but stricter clients refused to connect. Here's how we check for both problems by hand and with a small script.

See what the server is really sending

Connect and show the certificates the server sends. Always pass -servername so SNI picks the right certificate on servers that host more than one name. Without it, a shared server may hand you its default certificate, and the check looks fine while the real site is broken.

openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com -showcerts </dev/null

The output lists each certificate the server sent, numbered from 0 (the leaf). Each one shows s: (subject) and i: (issuer). A healthy chain from a public CA usually has the leaf at 0 and one or more intermediates after it. If only certificate 0 appears and its issuer is an intermediate CA, the server isn't sending its chain.

Reading the expiry date

Pipe the leaf into openssl x509 to read its dates:

echo | openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate

subject=CN = vpn.contoso.com
issuer=C = US, O = Example CA, CN = Example Intermediate R1
notAfter=Mar 10 23:59:59 2027 GMT

In scripts, I lean on -checkend SECONDS. It exits non-zero if the certificate expires within that many seconds, so you don't have to parse dates or deal with the differences between GNU and BSD date.

Checking the chain

Look at the verification result at the end of the s_client output:

openssl s_client -connect vpn.contoso.com:443 -servername vpn.contoso.com </dev/null 2>/dev/null | grep 'Verify return code'

You want Verify return code: 0 (ok). The failures we see most:

• 21 (unable to verify the first certificate) or 20 (unable to get local issuer certificate) usually means the server is missing an intermediate.
• 10 (certificate has expired) means what it says.
• 18 or 19 means a self-signed certificate, or a private CA this machine doesn't trust. For internal hosts, pass -CAfile with your root.

So why don't browsers complain about a missing intermediate? Many browsers cache intermediates or fetch them using the Authority Information Access URL in the leaf. Command-line tools, VPN clients, Java and Python applications, mail servers and older mobile devices often don't, so they fail while the browser looks fine. That's why I never test with a browser alone. The fix is to install the full chain on the server, meaning the leaf plus intermediates but not the root.

The script we run every morning

Put your endpoints in a text file, one host port per line:

vpn.contoso.com 443
mail.contoso.com 443
portal.contoso.com 8443
fw01.contoso.com 443

Then run this script from cron daily:

#!/usr/bin/env bash
# cert-check.sh - warn on certificates expiring soon or with broken chains
# usage: cert-check.sh hosts.txt [warn_days]
LIST="${1:-hosts.txt}"
WARN_DAYS="${2:-30}"
SECS=$((WARN_DAYS * 86400))
status=0

while read -r host port; do
  [[ -z "$host" || "$host" == \#* ]] && continue
  port="${port:-443}"
  out=$(timeout 15 openssl s_client -connect "$host:$port" -servername "$host" </dev/null 2>/dev/null)
  cert=$(printf '%s\n' "$out" | openssl x509 2>/dev/null)
  if [[ -z "$cert" ]]; then
    echo "ERROR  $host:$port  no certificate retrieved"; status=2; continue
  fi
  end=$(printf '%s\n' "$cert" | openssl x509 -noout -enddate | cut -d= -f2)
  verify=$(printf '%s\n' "$out" | grep -m1 'Verify return code' | sed 's/^ *//')
  if ! printf '%s\n' "$cert" | openssl x509 -noout -checkend 0 >/dev/null; then
    echo "EXPIRED $host:$port  $end"; status=2
  elif ! printf '%s\n' "$cert" | openssl x509 -noout -checkend "$SECS" >/dev/null; then
    echo "WARN   $host:$port  expires $end (under $WARN_DAYS days)"; status=1
  else
    echo "OK     $host:$port  expires $end"
  fi
  if [[ "$verify" != *"0 (ok)"* ]]; then
    echo "CHAIN  $host:$port  $verify"; [[ $status -lt 1 ]] && status=1
  fi
done < "$LIST"
exit $status

Schedule it so it only emails when the exit code is non-zero. That way, a certificate email in your inbox always means something needs attention.

0 7 * * * /opt/scripts/cert-check.sh /opt/scripts/hosts.txt 30 > /tmp/cert-check.out || mail -s 'Certificate check' noc@contoso.com < /tmp/cert-check.out

Most monitoring systems can do this natively too, with HTTP checks on certificate age or a TLS sensor. If you already have one, use it. We still find the script handy for appliances on odd ports and as a second opinion.

For us the hard part wasn't the script, it was building the list. VPN gateways, firewall admin pages, UPS cards and printers are the ones that expire unnoticed, so build the list from DNS and your firewall NAT rules rather than from memory.