TLS Triage

Someone reports “the site is broken” or “we get a certificate error”. Work through these in order. Each step either finds the problem or rules out a layer.

Examples use badssl.com, which hosts a deliberately broken site for nearly every TLS failure, and www.va.gov as a well-behaved public site. Everything here runs in Cygwin with the openssl, curl, bind-utils and jq packages.

1. Find the Failing Layer

Before opening openssl, run curl once and read the exit code. It separates DNS, TCP, and TLS failures without any guesswork.

curl -sS -o /dev/null -w '%{http_code}\n' https://expired.badssl.com
echo "exit code: $?"

Exit 60 covers four different problems. The message after the code tells you which one:

Do not reach for curl -k to make the error go away while triaging. It turns off exactly the check you are trying to run.

2. Read the Chain

-servername sends SNI. Without it, a server hosting many sites returns its default certificate, and you end up debugging the wrong certificate.

echo | openssl s_client -connect www.va.gov:443 -servername www.va.gov -showcerts 2>/dev/null \
  | grep -E '^ *[0-9] s:|^ +i:'

 0 s:CN=www.va.gov
   i:C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA DV R36
 1 s:C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA DV R36
   i:C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication Root R46
 2 s:C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication Root R46
   i:C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services

Read it top down. Each i: (issuer) should equal the next line’s s: (subject). If the list stops at 0 and the issuer is not a root your machine trusts, the server is missing its intermediate. That is the unable to get local issuer certificate case.

The leaf certificate’s names and dates:

echo | openssl s_client -connect www.va.gov:443 -servername www.va.gov 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

subject=CN=www.va.gov
issuer=C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA DV R36
notBefore=Dec 20 00:00:00 2025 GMT
notAfter=Jan 18 23:59:59 2027 GMT
X509v3 Subject Alternative Name:
    DNS:www.va.gov, DNS:va.gov

Clients ignore the CN and match only against the Subject Alternative Names. If the name you typed is not in that list, you get the hostname mismatch error.

For a pass/fail answer, -checkend exits non-zero if the certificate expires within the given number of seconds. That makes it usable in cron or CI:

echo | openssl s_client -connect www.va.gov:443 -servername www.va.gov 2>/dev/null \
  | openssl x509 -noout -checkend $((30 * 86400)) || echo "expires within 30 days"

3. Check Every IP

A name behind a load balancer or round-robin DNS resolves to several IPs, and a renewal that missed one node shows up as an intermittent error. This checks each IPv4 address separately, sending the same SNI to each:

#!/usr/bin/env bash
# Print the certificate expiry served by every IPv4 address behind a name.
# Usage: bash cert-expiry-by-ip.sh <hostname> [port]
set -euo pipefail

host=${1:?usage: $0 <hostname> [port]}
port=${2:-443}
now=$(date +%s)

# dig +short follows CNAMEs and prints them too; keep only the IPv4 answers.
dig +short A "$host" | grep -E '^[0-9.]+$' | while read -r ip; do
  end=$(timeout 5 openssl s_client -connect "$ip:$port" -servername "$host" </dev/null 2>/dev/null \
        | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2) || true
  if [ -z "$end" ]; then
    printf '%s,%s,NO CERTIFICATE\n' "$host" "$ip"
    continue
  fi
  days=$(( ($(date -d "$end" +%s) - now) / 86400 ))
  printf '%s,%s,%s,%s days\n' "$host" "$ip" "$end" "$days"
done

$ bash cert-expiry-by-ip.sh www.va.gov
www.va.gov,152.130.96.221,Jan 18 23:59:59 2027 GMT,115 days

dig +short prints CNAME targets before the addresses, which is why the script filters for lines that look like IPs.

4. Sweep a Domain

Certificate Transparency logs list every certificate a public CA has issued for a domain, which gives you a list of names nobody remembered to put in the inventory. This pulls the names from crt.sh, connects to each one in parallel, and reports days left plus whether the certificate actually verifies:

#!/usr/bin/env bash
# Find every name in Certificate Transparency logs for a domain, then check
# the certificate each one serves right now: days left, expiry, and whether
# it verifies. Soonest expiry first.
# Usage: bash cert-expiry-domain.sh <domain> [parallel-jobs]
#        subfinder -silent -d <domain> | bash cert-expiry-domain.sh - [parallel-jobs]
set -euo pipefail

domain=${1:?usage: $0 <domain> [parallel-jobs]}
jobs=${2:-10}

check() {
  local name=$1 out end days verify
  out=$(timeout 5 openssl s_client -connect "$name:443" -servername "$name" \
        -verify_hostname "$name" </dev/null 2>/dev/null) || true
  end=$(printf '%s' "$out" | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2) || true
  if [ -z "$end" ]; then
    printf 'NONE\t%s\tno TLS answer on 443\n' "$name"
    return
  fi
  verify=$(printf '%s' "$out" | sed -n 's/^ *Verify return code: [0-9]* (\(.*\))$/\1/p' | tail -1)
  days=$(( ($(date -d "$end" +%s) - $(date +%s)) / 86400 ))
  printf '%s\t%s\t%s\t%s\n' "$days" "$name" "$end" "$verify"
}
export -f check

names() {
  if [ "$domain" = - ]; then
    cat   # names on stdin, one per line
  else
    # name_value holds several names separated by newlines
    curl -fsS --retry 5 --retry-delay 10 --max-time 120 \
      "https://crt.sh/?q=%25.${domain}&output=json" | jq -r '.[].name_value'
  fi
}

# Wildcards are cut down to their base name so there is something to connect to.
names \
  | sed 's/^\*\.//' | tr 'A-Z' 'a-z' | sort -u \
  | xargs -P "$jobs" -I{} bash -c 'check "$1"' _ {} \
  | sort -n

Run it as bash cert-expiry-domain.sh badssl.com 20. Here it is fed a short list on stdin so the output is easy to read:

$ printf '%s\n' '*.badssl.com' expired.badssl.com self-signed.badssl.com \
    WRONG.host.badssl.com nope.invalid | bash cert-expiry-domain.sh - 5
-4183   expired.badssl.com      Apr 12 23:59:59 2015 GMT   certificate has expired
NONE    nope.invalid            no TLS answer on 443
31      badssl.com              Oct 26 20:03:01 2026 GMT   ok
31      wrong.host.badssl.com   Oct 26 20:03:01 2026 GMT   hostname mismatch
727     self-signed.badssl.com  Sep 21 21:00:16 2028 GMT   self-signed certificate

Things to know:

• crt.sh is a free service and often returns 502 under load. The script retries, then stops with curl’s error instead of printing an empty report. When it is down, feed the names in yourself: subfinder -silent -d badssl.com | bash cert-expiry-domain.sh -.
• An earlier version of this script only checked names that answered 200 OK first. HTTP/2 servers answer HTTP/2 200 and many sites answer with a redirect, so most live hosts were skipped. Connecting with openssl directly avoids the problem.
• NONE does not always mean dead. The name may be internal-only, behind a VPN, or listening only on a port other than 443.
• The days column only covers expiry. Read the last column too: a certificate with 700 days left that is self-signed is still broken.

5. Check Protocols

Which TLS versions does the server accept?

#!/usr/bin/env bash
# Which TLS versions does a server accept?
# Usage: bash tls-versions.sh <hostname> [port]
set -euo pipefail

host=${1:?usage: $0 <hostname> [port]}
port=${2:-443}

for v in tls1 tls1_1 tls1_2 tls1_3; do
  # SECLEVEL=0 lets the client offer old protocols at all. Distro crypto
  # policies can still block them; see the note under "Protocols and ciphers".
  if timeout 5 openssl s_client -connect "$host:$port" -servername "$host" \
       "-$v" -cipher 'DEFAULT:@SECLEVEL=0' </dev/null 2>/dev/null | grep -q '^New, TLS'; then
    printf '%-8s offered\n' "$v"
  else
    printf '%-8s not offered\n' "$v"
  fi
done

$ bash tls-versions.sh www.va.gov
tls1     not offered
tls1_1   not offered
tls1_2   offered
tls1_3   not offered

Your local OpenSSL limits what this can see. OpenSSL 3 refuses TLS 1.0 and 1.1 unless you lower the security level, which the script does with @SECLEVEL=0. Fedora and RHEL crypto policies can still block legacy signatures underneath that, and you get invalid digest instead of a handshake. So “not offered” for old versions can mean “my client could not ask”. For a compliance answer, use testssl.sh, which ships its own OpenSSL built with the legacy options, or nmap:

nmap --script ssl-enum-ciphers -p 443 www.va.gov

Both grade every cipher the server offers, which is what an audit against M-15-13 and BOD 18-01 needs.

6. Check HSTS

BOD 18-01 expects port 80 to redirect to HTTPS, and HTTPS responses to carry Strict-Transport-Security with max-age of at least one year, includeSubDomains, and preload. The check has to follow the redirect: va.gov and www.va.gov can send different headers.

#!/usr/bin/env bash
# Check HSTS on a host, and on wherever it redirects to, the way M-15-13 and
# BOD 18-01 grade it. Pass an IP to test one edge node or a CDN before DNS moves.
# Usage: bash hsts-check.sh <hostname> [ip]
set -euo pipefail

host=${1:?usage: $0 <hostname> [ip]}
ip=${2:-}

pin=()
[ -n "$ip" ] && pin=(--resolve "$host:443:$ip")

hsts() {  # print the HSTS header value of a URL, or nothing
  curl -sS -o /dev/null -D - --max-time 10 "${pin[@]}" "$1" \
    | tr -d '\r' | sed -n 's/^strict-transport-security: *//Ip' | tail -1
}

report() {
  local url=$1 got lc age missing=()
  got=$(hsts "$url")
  if [ -z "$got" ]; then
    echo "FAIL  $url sends no Strict-Transport-Security header"
    return
  fi
  lc=$(printf '%s' "$got" | tr 'A-Z' 'a-z')
  age=$(printf '%s' "$lc" | sed -n 's/.*max-age="\{0,1\}\([0-9]*\).*/\1/p')
  [ "${age:-0}" -ge 31536000 ] || missing+=("max-age>=31536000")
  [[ $lc == *includesubdomains* ]] || missing+=(includeSubDomains)
  [[ $lc == *preload* ]]           || missing+=(preload)
  if [ ${#missing[@]} -eq 0 ]; then
    echo "OK    $url  $got"
  else
    echo "WARN  $url  $got"
    echo "      missing: ${missing[*]}"
  fi
}

report "https://$host/"

# Only the first hop is pinned: --resolve applies to $host alone, so a
# redirect to another name resolves normally.
final=$(curl -sS -L -o /dev/null --max-time 15 "${pin[@]}" -w '%{url_effective}' "https://$host/")
[ "$final" != "https://$host/" ] && report "$final"

# Port 80 should only redirect to https.
loc=$(curl -sS -o /dev/null --max-time 10 ${ip:+--resolve "$host:80:$ip"} -w '%{http_code} %{redirect_url}' "http://$host/")
echo "INFO  http://$host/ -> $loc"

$ bash hsts-check.sh va.gov
WARN  https://va.gov/  max-age=31536000; preload
      missing: includeSubDomains
OK    https://www.va.gov/  max-age=31536000; includeSubDomains; preload
INFO  http://va.gov/ -> 301 https://va.gov:443/

The script checks each directive on its own, because servers write them in different orders and cases. GitHub, for example, sends includeSubdomains.

7. Test Before Cutover

Before a DNS cutover, you want to know the new origin or CDN edge serves the right certificate for your name. --resolve makes curl connect to an IP you choose while still sending the real hostname in SNI and the Host header:

curl -sS -o /dev/null --resolve www.va.gov:443:152.130.96.221 \
  -w 'ip=%{remote_ip} http=%{http_code} verify=%{ssl_verify_result}\n' https://www.va.gov/

The format is host:port:address, and the host and port must match the URL exactly. A malformed entry such as --resolve https://www.va.gov:443:1.2.3.4 at least fails loudly (exit 49). A well-formed entry that does not match is worse: --resolve va.gov:443:... for a www.va.gov URL, or port 80 for an https:// URL, is silently ignored, so curl resolves normally, tests the old site, and reports success. Always print %{remote_ip} to confirm you reached the IP you meant.

When the new target is a CDN hostname rather than an IP, --connect-to saves the dig step:

curl -sS -o /dev/null --connect-to www.example.gov:443:www.example.gov.edgekey.net:443 \
  -w 'ip=%{remote_ip} verify=%{ssl_verify_result}\n' https://www.example.gov/

verify=0 means the certificate on the new target is valid for your name. If it is not, curl exits 60 and prints a non-zero verify (an OpenSSL verify code, such as 20 for a missing issuer), and that is what your users would see on cutover day. This is the one test where -k would hide exactly the problem you are looking for.

hsts-check.sh above takes the IP as a second argument (bash hsts-check.sh www.va.gov 152.130.96.221) to check the headers the new target sends, too.

To see the cutover from your users’ point of view, run the same curl from several regions. A VPN with a CLI works, for example piactl set region uk-london && piactl connect in a loop. This matters most when you have users overseas.

8. Time the Request

When the complaint is “TLS is slow”, split the request into phases:

curl -sS -o /dev/null https://www.va.gov/ -w \
  'dns=%{time_namelookup}s tcp=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n'

dns=0.000012s tcp=0.009375s tls=0.036600s ttfb=0.390449s total=0.523465s

Each value is cumulative from the start of the request. So the TLS handshake took tls - tcp, about 27 ms here, and the server spent ttfb - tls thinking. A large tls - tcp next to a small tcp usually points to a long certificate chain, a busy server doing expensive key exchanges, or a TLS-inspecting proxy in the path. See Latency for why the tcp number cannot go below the round trip.

Further Reading

• badssl.com: a broken example for most failure modes, useful for checking that your client really does fail
• SSL Labs deployment best practices
• Feisty Duck OpenSSL Cookbook