cryptography-certificate-management

Certificate lifecycle management including rotation, renewal, monitoring, automation with Let's Encrypt and ACME

Certificate Management

Scope: Certificate rotation, renewal, monitoring, Let's Encrypt, ACME protocol, automation Lines: ~350 Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)

When to Use This Skill

Activate this skill when:

Certificate Lifecycle

1. Generation → Create private key and CSR
2. Issuance  → CA signs certificate
3. Deployment → Install on servers
4. Monitoring → Track expiration
5. Renewal   → Get new certificate before expiry
6. Rotation  → Replace old with new
7. Revocation → Invalidate if compromised (optional)

Let's Encrypt & ACME

Overview

Let's Encrypt:

ACME Protocol:

1. Account creation
2. Domain validation (HTTP-01, DNS-01, TLS-ALPN-01)
3. Certificate request
4. Certificate issuance
5. Certificate renewal

Certbot

Installation:

# Ubuntu/Debian
sudo apt install certbot python3-certbot-nginx

# macOS
brew install certbot

HTTP-01 Challenge (webroot):

# Obtain certificate
sudo certbot certonly --webroot \
    -w /var/www/html \
    -d example.com \
    -d www.example.com

# Certificates stored in:
# /etc/letsencrypt/live/example.com/fullchain.pem
# /etc/letsencrypt/live/example.com/privkey.pem

Nginx plugin (automatic):

# Obtain and install certificate
sudo certbot --nginx -d example.com -d www.example.com

# Certbot automatically:
# - Validates domain
# - Obtains certificate
# - Updates nginx config
# - Reloads nginx

DNS-01 Challenge (wildcards):

# For wildcard certificates
sudo certbot certonly --manual \
    --preferred-challenges dns \
    -d '*.example.com' \
    -d example.com

# Add TXT record to DNS:
# _acme-challenge.example.com → [provided value]

Auto-renewal:

# Test renewal
sudo certbot renew --dry-run

# Setup cron (or systemd timer)
0 0,12 * * * certbot renew --quiet --post-hook "systemctl reload nginx"

acme.sh

Alternative ACME client:

# Install
⚠️ **SECURITY**: Piping curl to shell is dangerous. For production:

Download script first

curl -O https://get.acme.sh

Verify checksum

sha256sum get.acme.sh

Review content

less get.acme.sh

Then execute

bash get.acme.sh

For development/learning only:

curl https://get.acme.sh | sh

Issue certificate (HTTP validation)

acme.sh --issue -d example.com -w /var/www/html

Issue wildcard (DNS validation with Cloudflare)

export CF_Token="your-cloudflare-api-token" # Placeholder - replace with actual API token from Cloudflare dashboard acme.sh --issue --dns dns_cf -d example.com -d '*.example.com'

Install certificate

acme.sh --install-cert -d example.com \ --key-file /etc/ssl/private/example.com.key \ --fullchain-file /etc/ssl/certs/example.com.crt \ --reloadcmd "systemctl reload nginx"

Auto-renewal (automatic after install)

acme.sh --cron


---

## Automation Patterns

### Pattern 1: Systemd Timer for Renewal

**Service file** (`/etc/systemd/system/certbot-renew.service`):

[Unit] Description=Certbot Renewal

[Service] Type=oneshot ExecStart=/usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"


**Timer file** (`/etc/systemd/system/certbot-renew.timer`):

[Unit] Description=Certbot Renewal Timer

[Timer] OnCalendar=daily RandomizedDelaySec=1h Persistent=true

[Install] WantedBy=timers.target


**Enable**:

sudo systemctl enable --now certbot-renew.timer sudo systemctl list-timers certbot-renew.timer


### Pattern 2: Kubernetes cert-manager

**Install cert-manager**:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml


**ClusterIssuer** (Let's Encrypt):

apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-prod solvers:

ingress: class: nginx


**Certificate resource**:

apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: example-com-tls namespace: default spec: secretName: example-com-tls issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames:


**Ingress annotation** (automatic):

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example annotations: cert-manager.io/cluster-issuer: "letsencrypt-prod" spec: tls:

secretName: example-com-tls rules:

http: paths:

pathType: Prefix backend: service: name: example port: number: 80


### Pattern 3: Custom Automation Script

**Python renewal script**:

#!/usr/bin/env python3 import subprocess import ssl import socket from datetime import datetime, timedelta import smtplib from email.message import EmailMessage

def check_certificate_expiry(hostname, port=443): """Check days until certificate expires""" context = ssl.create_default_context() with socket.create_connection((hostname, port)) as sock: with context.wrap_socket(sock, server_hostname=hostname) as ssock: cert = ssock.getpeercert() not_after = datetime.strptime(cert['notAfter'], '%b %d %H:%M:%S %Y %Z') days_remaining = (not_after - datetime.now()).days return days_remaining

def renew_certificate(domain): """Renew certificate using certbot""" result = subprocess.run( ['certbot', 'renew', '--cert-name', domain, '--quiet'], capture_output=True, text=True ) return result.returncode == 0

def send_alert(subject, body): """Send email alert""" msg = EmailMessage() msg['Subject'] = subject msg['From'] = 'certbot@example.com' msg['To'] = 'admin@example.com' msg.set_content(body)

with smtplib.SMTP('localhost') as s: s.send_message(msg)

def main(): domains = ['example.com', 'api.example.com']

for domain in domains: try: days = check_certificate_expiry(domain) print(f"{domain}: {days} days until expiry")

# Renew if < 30 days remaining if days < 30: print(f"Renewing {domain}...") if renew_certificate(domain): send_alert( f"Certificate renewed: {domain}", f"Successfully renewed certificate for {domain}" ) else: send_alert( f"Certificate renewal FAILED: {domain}", f"Failed to renew certificate for {domain}" )

# Alert if < 14 days and renewal failed elif days < 14: send_alert( f"Certificate expiring soon: {domain}", f"Certificate for {domain} expires in {days} days" )

except Exception as e: send_alert( f"Certificate check failed: {domain}", f"Error checking {domain}: {str(e)}" )

if __name__ == '__main__': main()


---

## Monitoring

### Check Expiration

**OpenSSL**:

Check expiration date

echo | openssl s_client -connect example.com:443 2>/dev/null | \ openssl x509 -noout -dates

Days until expiry

echo | openssl s_client -connect example.com:443 2>/dev/null | \ openssl x509 -noout -enddate | \ awk -F= '{print $2}' | xargs -I {} date -d {} +%s | \ awk -v now=$(date +%s) '{print int(($1-now)/86400)}'


**Prometheus Exporter**:

ssl_exporter

apiVersion: v1 kind: Service metadata: name: ssl-exporter spec: ports:

selector: app: ssl-exporter --- apiVersion: apps/v1 kind: Deployment metadata: name: ssl-exporter spec: template: spec: containers:

image: ribbybibby/ssl-exporter:latest args:


**Prometheus alert**:

groups:

rules:

expr: ssl_cert_not_after - time() < 86400 * 30 labels: severity: warning annotations: summary: "Certificate expiring in < 30 days" description: "Certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"


### Monitoring Services

**SSL Labs API**:

import requests import time

def check_ssl_labs(domain): """Check SSL configuration with SSL Labs""" api_url = "https://api.ssllabs.com/api/v3/analyze"

# Start scan requests.get(api_url, params={'host': domain, 'startNew': 'on'})

# Poll for results while True: resp = requests.get(api_url, params={'host': domain}).json() if resp['status'] == 'READY': grade = resp['endpoints'][0]['grade'] return grade time.sleep(30)

grade = check_ssl_labs('example.com') print(f"SSL Labs grade: {grade}")


---

## Certificate Rotation

### Zero-Downtime Rotation

**Strategy**:
  1. Obtain new certificate
  2. Deploy new certificate alongside old
  3. Update configuration to use new
  4. Reload/restart services
  5. Verify new certificate active
  6. Remove old certificate

**Nginx example**:

1. Obtain new certificate

certbot certonly --webroot -w /var/www/html -d example.com

2. Test configuration

nginx -t

3. Reload nginx (no downtime)

systemctl reload nginx

4. Verify

echo | openssl s_client -connect example.com:443 2>/dev/null | \ openssl x509 -noout -dates


### Multi-Server Rotation

**Ansible playbook**:

hosts: webservers tasks:

copy: src: "{{ item }}" dest: "/etc/ssl/certs/" mode: 0644 loop:

copy: src: example.com.key dest: /etc/ssl/private/ mode: 0600

systemd: name: nginx state: reloaded

shell: | echo | openssl s_client -connect localhost:443 2>/dev/null | \ openssl x509 -noout -dates register: cert_check changed_when: false

debug: var: cert_check.stdout


---

## Best Practices

### 1. Renewal Window

✅ Good: Renew 30 days before expiry

if [ $days_remaining -lt 30 ]; then renew_certificate fi

❌ Bad: Wait until last minute

if [ $days_remaining -lt 3 ]; then renew_certificate fi


### 2. Monitoring

✅ Good: Multiple monitoring approaches

❌ Bad: No monitoring

(certificate expires, site goes down)


### 3. Backup Certificates

✅ Good: Backup before rotation

cp /etc/ssl/certs/example.com.crt /backup/ certbot renew

❌ Bad: No backup

certbot renew

(if renewal fails, no fallback)


---

## Troubleshooting

### Issue 1: Rate Limits

**Let's Encrypt limits**:

**Solution**: Use staging environment for testing

certbot --staging -d example.com


### Issue 2: Renewal Failure

**Check renewal status**:

certbot certificates


**Common issues**:
- Webroot path wrong
- DNS not propagated (DNS-01)
- Firewall blocking (HTTP-01)
- Rate limit exceeded

**Debug**:

certbot renew --dry-run --debug


---

## Related Skills

- `cryptography-pki-fundamentals` - Certificate basics
- `cryptography-tls-configuration` - TLS setup
- `infrastructure-automation` - Deployment automation
- `observability-metrics-monitoring` - Monitoring setup

---

## Level 3: Resources

**Location**: `~/.hanzo/skills/cryptography/certificate-management/resources/`

This skill includes comprehensive Level 3 resources for production certificate management implementations:

### REFERENCE.md (~3,100 lines)

Comprehensive technical reference covering:
- **Fundamentals**: Digital certificates, X.509 standard, trust chains, PKI concepts
- **Certificate Components**: Subject, SANs, extensions, key usage, certificate formats (PEM, DER, PKCS#12)
- **Certificate Lifecycle**: Generation, CSR creation, issuance, deployment, monitoring, renewal, rotation, revocation
- **CA Hierarchies**: Root/intermediate CAs, public vs private CAs, trust models
- **ACME Protocol**: Challenge types (HTTP-01, DNS-01, TLS-ALPN-01), ACME flow, client implementations
- **Let's Encrypt**: Rate limits, staging environment, trust chain, wildcard certificates
- **Certificate Automation**: Certbot usage, acme.sh, renewal hooks, systemd timers
- **Kubernetes cert-manager**: Installation, issuers, certificate resources, ingress annotations
- **Mutual TLS (mTLS)**: Server/client configuration, service mesh integration, zero-trust patterns
- **Certificate Revocation**: CRL, OCSP, OCSP stapling, must-staple extension
- **Certificate Transparency**: CT logs, monitoring, unauthorized certificate detection
- **Certificate Pinning**: Public key pinning, HPKP (deprecated), mobile implementations
- **Monitoring**: Prometheus exporters, expiration tracking, alerting strategies
- **Best Practices**: Lifetime management, key protection, automation, testing, backup/recovery
- **Common Issues**: Troubleshooting expired certs, chain issues, key mismatches, SAN errors
- **Security**: Private key protection, validation requirements, weak cryptography avoidance
- **Compliance**: PCI-DSS, HIPAA, SOC2, GDPR certificate requirements
- **Tools**: OpenSSL, certbot, acme.sh, step CLI, cfssl, cert-manager, SSL Labs, testssl.sh

### Scripts (3 production-ready tools)

**validate_certificates.py** (580+ lines) - Comprehensive certificate validation
- Validates certificates from remote hosts or local files
- Checks expiration dates with configurable thresholds
- Detects weak algorithms (MD5, SHA-1, weak RSA keys)
- Validates certificate chains and hostname matching
- SAN validation including wildcard matching
- OCSP/CRL revocation checking support
- Compliance validation (PCI-DSS, HIPAA, SOC2)
- Batch validation from hosts file
- JSON output for CI/CD integration
- Detailed reporting with severity levels
- Usage: `./validate_certificates.py --host example.com --compliance PCI-DSS --json`

**renew_certificates.py** (450+ lines) - Automated ACME certificate renewal
- ACME protocol automation (Let's Encrypt, ZeroSSL)
- HTTP-01 and DNS-01 challenge support
- Automatic renewal based on expiration thresholds
- Pre/post/deploy hook support
- Automatic backup before renewal
- Rollback on renewal failure
- Batch renewal of all expiring certificates
- Dry-run mode for testing
- Staging environment support
- Custom ACME provider support
- DNS provider integration (Cloudflare, Route53, etc.)
- Usage: `./renew_certificates.py --domain example.com --dns-provider cloudflare --json`

**monitor_cert_expiry.sh** (550+ lines) - Certificate expiration monitoring
- Monitors certificates from remote hosts or local files
- Configurable warning/critical thresholds
- Multiple output formats (text, JSON, Prometheus, CSV)
- Batch monitoring from hosts file
- Directory scanning for certificate files
- Email/webhook/Slack alerting
- Certificate inventory export
- OCSP revocation status checking
- Prometheus metrics format for monitoring integration
- Color-coded terminal output
- Exit codes for CI/CD integration
- Usage: `./monitor_cert_expiry.sh --hosts-file hosts.txt --format prometheus --alert-email admin@example.com`

### Examples (6 production-ready implementations)

**python/acme_automation.py** - ACME protocol automation from scratch
- Account registration and key management
- Certificate ordering with ACME protocol
- HTTP-01 challenge implementation
- DNS-01 challenge for wildcard certificates
- CSR generation with multiple SANs
- Complete ACME workflow demonstration
- Staging environment support
- Production-ready patterns

**python/mtls_server_client.py** - Mutual TLS implementation
- mTLS server with client certificate verification
- mTLS client with certificate authentication
- Certificate generation helper
- TLS version and cipher suite configuration
- Client certificate extraction and validation
- Zero-trust authentication pattern
- Service-to-service authentication example

**python/prometheus_cert_exporter.py** - Prometheus metrics exporter
- Exports certificate expiration metrics
- Multi-host monitoring support
- Prometheus-compatible metrics format
- Certificate metadata collection
- Expiration time tracking
- Validity status monitoring
- Error counting and duration tracking
- Health check endpoint
- Production-ready monitoring integration

**kubernetes/cert-manager-setup.yaml** - Comprehensive cert-manager configuration
- ClusterIssuers for Let's Encrypt (production and staging)
- DNS-01 solvers (Cloudflare, Route53)
- Certificate resources with SANs
- Wildcard certificate examples
- Ingress annotation automation
- Namespace-scoped issuers
- Private CA issuer configuration
- mTLS client certificates
- Certificate lifecycle management
- Renewal configuration
- Troubleshooting helpers

**bash/certbot-automation.sh** - Complete certbot automation suite
- HTTP-01 webroot validation
- Nginx plugin integration
- Wildcard certificates (manual and automated DNS)
- DNS provider integration (Cloudflare, Route53)
- Systemd timer setup for auto-renewal
- Pre/post/deploy hooks
- Certificate revocation
- Force renewal
- Staging environment testing
- Custom CSR usage
- Certificate monitoring
- Nginx installation automation
- 18+ production examples

**config/nginx-ssl-best-practices.conf** - Production Nginx SSL configuration
- Modern TLS 1.2/1.3 configuration
- Strong cipher suite selection
- OCSP stapling setup
- HSTS and security headers
- Let's Encrypt integration
- HTTP to HTTPS redirect
- Wildcard certificate support
- mTLS configuration
- Multiple domain examples
- SSL Labs A+ rating configuration
- Certificate renewal integration
- PHP-FPM and reverse proxy examples

### Quick Start

Validate remote certificate

cd ~/.hanzo/skills/cryptography/certificate-management/resources/scripts ./validate_certificates.py --host example.com --check-revocation

Renew certificate automatically

./renew_certificates.py --domain example.com --webroot /var/www/html --post-hook "systemctl reload nginx"

Monitor certificate expiration

./monitor_cert_expiry.sh --hosts-file hosts.txt --threshold-warning 30 --json

Export Prometheus metrics

./monitor_cert_expiry.sh --hosts-file hosts.txt --format prometheus > /var/lib/prometheus/cert_metrics.prom

Run Python examples

cd ../examples/python pip install cryptography acme josepy prometheus_client python acme_automation.py http01 python mtls_server_client.py setup && python mtls_server_client.py server python prometheus_cert_exporter.py --hosts example.com:443

View comprehensive reference

cd ../ less REFERENCE.md


### Integration Notes

**CI/CD Integration**:

.github/workflows/certificate-check.yml

run: | ./scripts/validate_certificates.py \ --batch-file production-hosts.txt \ --compliance PCI-DSS \ --json \ --check-revocation


**Monitoring Setup**:

Prometheus scrape config

scrape_configs:

static_configs:

scrape_interval: 60s


**Alerting**:

Prometheus alert rules

expr: ssl_certificate_expiry_seconds < 86400 * 30 labels: severity: warning annotations: summary: "Certificate expiring in < 30 days"


**Kubernetes Deployment**:

Install cert-manager

kubectl apply -f examples/kubernetes/cert-manager-setup.yaml

Check certificate status

kubectl get certificates -A kubectl describe certificate example-com-tls


---

**Last Updated**: 2025-10-27