Build a Certificate Transparency Monitoring Pipeline with crt.sh and the CT Logs API (Hands-On Walkthrough)

{
“@context”: “https://schema.org”,
“@type”: “TechArticle”,
“headline”: “Build a Certificate Transparency Monitoring Pipeline with crt.sh and the CT Logs API (Hands-On Walkthrough)”,
“description”: “Learn to build a certificate transparency monitoring pipeline with crt.sh and the CT Logs API. Hands-on commands, JSON queries, and alerting for subdomain tracking.”,
“author”: {“@type”: “Organization”, “name”: “Hmmnm”},
“publisher”: {“@type”: “Organization”, “name”: “Hmmnm – Cybersecurity Tutorials”}
}

Certificate Transparency monitoring with crt.sh and the CT Logs API gives blue teams near–real-time visibility into every TLS certificate issued for their domains—including rogue certificates and forgotten subdomains that never made it into an asset inventory. In this walkthrough, you’ll build a working pipeline that polls CT logs, diffs new issuances against a baseline, and fires alerts to Slack or email before attackers can weaponize a misissuance you never saw.

TL;DR: A CT monitoring pipeline polls Certificate Transparency logs via crt.sh and the CT Logs API to catch new certificates and subdomains for your domains in near real time

If you own a domain, you are protected by a public, append-only ledger of every certificate ever issued for it. This pipeline turns that ledger into an early-warning system: defenders get alerted to misissuance and shadow infrastructure within minutes-to-hours; attackers lose the asymmetric advantage of knowing your attack surface better than you do.

Why Certificate Transparency Monitoring Matters for Blue Teams

Certificate Transparency (CT) came out of repeated CA compromise scandals—the 2011 DigiNotar breach being the canonical example—where certificates were issued for domains without the owner’s knowledge. Since April 2018, Chrome has required all newly issued certificates to appear in CT logs; Mozilla and Apple enforce similar policies. The result: every legitimate CA issuance is publicly logged, and misissuance is detectable.

That’s the defensive story. The offensive story is the same data, flipped. CT logs are the single richest source of passive subdomain enumeration available. An attacker querying crt.sh for %.example.com gets years of issuance history—internal hostnames, staging environments, and deprecated services included. Unlike DNS brute-forcing, CT queries are silent: no packets ever touch your infrastructure. When Cert Spotter and similar services surfaced this problem years ago, the conclusion was unavoidable—if a subdomain has ever had a certificate, assume attackers know it exists.

The blue team playbook, then, is straightforward:

• Catch misissuance: a certificate from a CA you’ve never used is a red flag worth escalating immediately.
• Recover forgotten hostnames: CT logs routinely surface staging, dev, and VPN endpoints missing from CMDBs.
• Detect lookalike domains: issuances on typosquats near your brand (e.g., exarnple.com) show up in the same logs.
• Validate CA hygiene: confirm all certificates come from your approved CAs and account for every issuance.

CISA and OWASP both flag incomplete asset inventories as a root cause of breaches—CT monitoring is one of the cheapest inventory feeds you can wire up. See also our guides on subdomain takeover and TLS misconfiguration for what happens when this data is left unmonitored.

How CT Logs, crt.sh, and the CT Logs API Fit Together

Three distinct layers, and the distinction matters:

• CT log servers are the source of truth. Operators like Google (ct.googleapis.com endpoints such as argon2025, rocketeer2025), DigiCert, and Sectigo run cryptographically verifiable, append-only Merkle tree logs. CAs submit every certificate they issue to at least two approved logs (per Chrome’s CT policy).
• The CT Logs API is the RFC 6962 protocol those log servers expose: get-sth (signed tree head), get-entries (raw log entries), get-roots (accepted root certificates). Talking to it directly gives you raw, base64-encoded entries—complete but unfriendly.
• crt.sh is Sectigo’s aggregate search front-end. It continuously harvests all public CT logs, stores certificates in a PostgreSQL database, and exposes a simple query interface with JSON output. It’s not a log—it’s an indexer over all logs.

Practical rule: use crt.sh for discovery and monitoring (one query covers every log), and the CT Logs API when you need cryptographic guarantees—raw entries, inclusion proofs, or validation against a signed tree head. Monitors like Cert Spotter do the latter rigorously; your homegrown pipeline will lean on crt.sh and drop to the raw API when needed.

Prerequisites and Lab Setup

• curl and jq for quick queries and JSON filtering.
• Python 3.8+ with requests (pip install requests).
• A test domain you control—issue a certificate for a throwaway subdomain so you can watch the pipeline fire end-to-end.
• cron or a systemd timer for scheduling.
• Optional: a Slack incoming webhook URL or SMTP relay for alerting.

Verify your tooling:

curl --version && jq --version && python3 --version

Querying crt.sh for Issued Certificates

The core endpoint takes a query and an output format. To pull every certificate ever issued for any subdomain of your domain, as JSON:

curl -s "https://crt.sh/?q=%25.example.com&output=json" | jq '.[0:3]'

The %25 is the URL-encoded wildcard %—this matches all subdomains. Sample output (trimmed):

[
  {
    "issuer_ca_id": 164,
    "issuer_name": "C=US, O=Let's Encrypt, CN=R11",
    "common_name": "staging.example.com",
    "name_value": "staging.example.comnapi-staging.example.com",
    "id": "12847392014",
    "entry_timestamp": "2025-01-14T08:22:31.117",
    "not_before": "2025-01-14T07:22:30",
    "not_after": "2025-04-14T07:22:29",
    "serial_number": "04a1f2..."
  }
]

Two fields earn their keep: name_value, which contains every SAN on the certificate (newline-separated—this is where forgotten hostnames surface), and entry_timestamp, which is when the cert hit the log. A more targeted query for recent Let’s Encrypt issuances:

curl -s "https://crt.sh/?q=%25.example.com&output=json" 
  | jq '[.[] | select(.issuer_name | contains("Let")) | {common_name, not_before}]'

You can also query by exact hostname (no wildcard), by serial number, or by SHA-256 fingerprint. Note that crt.sh has no API key and no documented rate limit—but it’s a free community service backed by a real database. Hammer it and you’ll be rate-limited or worse. Poll on a schedule; don’t loop.

Pulling Entries Directly from the CT Logs API (get-entries, get-roots)

When you need raw entries, the log server API at ct.googleapis.com speaks RFC 6962. First, check the log’s size and current state:

curl -s https://ct.googleapis.com/rocketeer2025/ct/v1/get-sth | jq

Fetch the accepted root certificates:

curl -s https://ct.googleapis.com/rocketeer2025/ct/v1/get-roots | jq '.certificates | length'

And pull a batch of entries by index range:

curl -s "https://ct.googleapis.com/rocketeer2025/ct/v1/get-entries?start=0&end=1" | jq '.entries[0].leaf_input[:40]'

Everything comes back base64-encoded—both leaf_input (a MerkleTreeLeaf structure wrapping a TimestampedEntry) and extra_data. Decoding is where most walkthroughs fall over, because the output is DER, not plain text. Here’s how to decode base64 entries returned by get-entries (the same logic applies to the JSON-formatted get-rows endpoint on log mirrors):

import base64, requests

resp = requests.get(
    "https://ct.googleapis.com/rocketeer2025/ct/v1/get-entries",
    params={"start": 0, "end": 0}, timeout=30
).json()

for entry in resp["entries"]:
    leaf = base64.b64decode(entry["leaf_input"])
    # Parse the TLSH structured leaf with the 'ct' library:
    # pip install ctutl  (or use cryptography + manual TLS parsing)
    from ctutl import get_certificate_from_trillian_leaf
    cert = get_certificate_from_trillian_leaf(leaf)
    print(cert.subject)

Merkle trees in plain English: the log is a binary hash tree. Every leaf is a hash of one certificate entry; every parent is a hash of its children; the root—the signed tree head—is a single hash committing to the entire contents. Because of this structure, a log operator can prove any individual certificate is included without revealing the whole log, and can’t quietly remove or alter an old entry without breaking every hash above it. Append-only isn’t a policy promise; it’s mathematics. That’s why you can’t remove a hostname from CT (more on that in the FAQ).

Building the Monitoring Script

The pipeline logic is a diff: fetch current state, compare against stored state, emit the delta. Full working script:

#!/usr/bin/env python3
"""CT monitor: diff crt.sh issuances for watched domains against a baseline."""
import json, time, pathlib, requests, sys

DOMAINS = ["example.com", "example.net"]
STATE_FILE = pathlib.Path("ct_state.json")
KNOWN_ISSUERS = {"Let's Encrypt", "DigiCert", "Sectigo"}

def fetch(domain):
    for attempt in range(3):
        try:
            r = requests.get(
                "https://crt.sh/",
                params={"q": f"%.{domain}", "output": "json"},
                timeout=60,
            )
            if r.status_code == 200:
                return r.json()
        except requests.RequestException:
            pass
        time.sleep(2 ** attempt * 5)  # exponential backoff
    sys.exit(f"crt.sh query failed for {domain}")

def main():
    state = json.loads(STATE_FILE.read_text()) if STATE_FILE.exists() else {}
    seen = set(state.get("cert_ids", []))
    current = {}
    new_findings = []

    for domain in DOMAINS:
        for cert in fetch(domain):
            cid = cert["id"]
            current[cid] = True
            if cid in seen:
                continue
            hosts = cert.get("name_value", cert.get("common_name", "")).split("n")
            issuer_ok = any(k in cert.get("issuer_name", "") for k in KNOWN_ISSUERS)
            new_findings.append({
                "domain": domain,
                "hosts": hosts,
                "issuer": cert["issuer_name"],
                "not_before": cert["not_before"],
                "unexpected_ca": not issuer_ok,
            })

    if new_findings:
        print(json.dumps(new_findings, indent=2))
        send_alert(new_findings)  # defined in the next section

    state["cert_ids"] = list(current.keys())
    state["updated"] = time.time()
    STATE_FILE.write_text(json.dumps(state))

if __name__ == "__main__":
    main()

Design notes: filter by not_before if you only care about fresh issuances (CT logs include historical entries; your first run seeds the baseline, and everything before it is history). Filter on issuer_name to flag certificates from unapproved CAs. Store state as a flat JSON file for simplicity—or SQLite if you’re tracking dozens of domains and want queryable history.

Alerting: Email, Slack, and Webhook Notifications

The decision to page a human should be deliberate: new subdomains are informational, unexpected CAs are incidents. Wire up a Slack webhook:

import os

def send_alert(findings):
    webhook = os.environ["SLACK_WEBHOOK_URL"]
    for f in findings:
        sev = ":rotating_light: UNEXPECTED CA" if f["unexpected_ca"] else ":new: New cert"
        text = f"{sev} — {', '.join(f['hosts'])} issued by {f['issuer']} (not_before {f['not_before']})"
        requests.post(webhook, json={"text": text}, timeout=10)

    # Escalate unexpected CAs via email as well
    ca_alerts = [f for f in findings if f["unexpected_ca"]]
    if ca_alerts:
        # pipe to your mail relay, e.g. subprocess msmtp/sendmail, or SMTP via smtplib
        pass

For generic webhook consumers (PagerDuty, Splunk, a SIEM collector), POST the same JSON body and let downstream routing decide severity. Treat unexpected-CA events as security incidents per your IR process, not as certificate-op tickets—misissuance combined with an unnoticed subdomain is a full impersonation setup.

Scheduling and Hardening the Pipeline

Run it via cron every 15–30 minutes:

*/30 * * * * /usr/local/bin/ct_monitor.py >> /var/log/ct_monitor.log 2>&1

Or, preferably, a systemd timer with jitter and a hard timeout:

[Unit]
Description=CT monitoring run

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ct_monitor.py
TimeoutStartSec=300
Environment=SLACK_WEBHOOK_URL=/etc/ct_monitor/webhook

[Timer]
OnCalendar=*:0/30
RandomizedDelaySec=120
Persistent=true

[Install]
WantedBy=timers.target

Hardening checklist:

• Retry with exponential backoff—already in the script above; never hammer crt.sh in a tight loop.
• Respect the service: crt.sh is free. Polling intervals under 5–10 minutes are antisocial and will get you throttled.
• Log every run—including failures. A silently dead monitor is worse than none.
• Protect the state file and webhook secret: restrictive file permissions; never commit the webhook URL.
• Handle schema drift: crt.sh occasionally returns HTML error pages or truncated results under load. Validate that the response parses as a JSON list before diffing.
• Set a maximum state-file growth strategy: prune certificate IDs older than a couple of years.

CTF Applications: Subdomain Recon with CT Logs

If you play CTFs, crt.sh is a first-move OSINT tool. Given only a domain in a recon challenge, the fastest passive enumeration is:

curl -s "https://crt.sh/?q=%25.target-domain.com&output=json" 
  | jq -r '.[].name_value' | tr 'n' 'n' | sort -u

That one-liner extracts every SAN from every certificate ever issued—often revealing staging hosts, admin panels, and the exact subdomain holding the flag. Pair it with DNS resolution and HTTP probing (httpx, httprobe) to find live hosts, and you’ve built the standard CT-based recon workflow in under a minute. Practice this on your own infrastructure: if the output surprises you, it will surprise an attacker less.

Limitations and Alternatives

Honest constraints of this pipeline:

• Latency: CT logs update continuously, but crt.sh indexing can lag minutes-to-hours. If you need guaranteed ingestion SLAs, monitor logs directly via the RFC 6962 API against the signed tree head.
• No cryptographic verification: crt.sh gives you search results, not Merkle inclusion proofs. Tools like Cert Spotter (SSLMate) verify STHs properly.
• Availability: crt.sh is a shared community service with no SLA. Don’t build production alerting that depends solely on it.

Managed alternatives worth knowing: Cert Spotter, Facebook’s Certificate Transparency Monitoring tooling, Cloudflare’s Mercury, and commercial CT monitoring built into Censys, Shodan, and various attack-surface-management platforms. If you’re self-hosting, the google/certificate-transparency Go tools let you run a log monitor against the raw logs. For most teams, though, a 100-line script on a cron job covers 90% of the value—start there, and graduate to a verified monitor when compliance or threat model demands it.

Frequently Asked Questions

Is crt.sh an official CT log?

No. crt.sh is a search interface operated by Sectigo that aggregates and indexes all public CT logs; the official logs themselves are append-only services run by organizations like Google, DigiCert, and Sectigo. crt.sh is a consumer of that data, not a log.

How often are CT logs updated?

The underlying logs update continuously—CAs submit within seconds of issuance—but crt.sh’s indexing can lag behind. Poll on a schedule and expect minutes-to-hours of delay between issuance and visibility in your pipeline.

Can I delete a subdomain from Certificate Transparency logs?

No. CT logs are append-only and public by design—the Merkle tree structure makes retroactive removal cryptographically impossible. Plan hostname disclosure accordingly: don’t issue certificates for hostnames whose existence you want to keep secret.

Do I need an API key for crt.sh or the CT Logs API?

No keys are required for either. Both are open, but respect rate limits: use reasonable polling intervals (15–30 minutes is plenty), implement backoff, and remember crt.sh is a free community service.

Related reading

• Hands-On OpenSSL Lab: Generate, Inspect and Troubleshoot TLS Certificates with Real Commands
• Build a Ransomware Sandbox with FLARE-VM and CAPE: Automated Config Extraction and IOC Dumping

Published by Hmmnm

Hands-on cybersecurity tutorials, CVE breakdowns, and guided learning paths — written and lab-tested by the Hmmnm team.

Learning Paths · About Hmmnm · Editorial policy