VayuPress Operations Runbook

Version: 1.0.0 (Prompt 6 — Operations)
Audience: Production operators, on-call maintainers
SLA: 99.9% uptime target; P1 response < 15 minutes

Quick Reference

# Service status
systemctl status vayupress meilisearch isso

# Health check
curl -s http://localhost:8080/health | jq .
curl -s http://localhost:8080/health/dependencies | jq .

# Logs (last 100 lines)
journalctl -u vayupress -n 100 --no-pager

# Restart application
systemctl restart vayupress

# Emergency stop
systemctl stop vayupress

Service Architecture

Health Endpoints

Runbooks

RB-01: Application Not Starting

1. Check logs: journalctl -u vayupress -n 50
2. Look for CHECKSUM DRIFT — means a migration was tampered (ADR-0034). Do NOT restart until investigated.
3. Look for FATAL: database locked — kill any zombie vayupress processes: pkill -f vayupress
4. Check data dir permissions: ls -la /data/vayupress.db
5. Verify env vars loaded: systemctl cat vayupress | grep EnvironmentFile
6. Test binary directly: sudo -u vayupress /usr/local/bin/vayupress --dry-run

RB-02: High Memory Usage (>800 MB)

1. Check current memory: systemctl status vayupress | grep Memory
2. Check for write queue backlog: curl http://localhost:8080/health/dependencies | jq .queue
3. Check WAL size: ls -lh /data/vayupress.db-wal
4. If WAL > 100 MB, checkpoint manually:

   sqlite3 /data/vayupress.db "PRAGMA wal_checkpoint(RESTART);"
   

5. If memory continues to grow, restart: systemctl restart vayupress
6. If recurring: check for runaway pprof goroutine, review recent deploys

RB-03: Database Corruption

1. Run integrity check: sqlite3 /data/vayupress.db "PRAGMA integrity_check;"
2. If errors, IMMEDIATELY stop writes: systemctl stop vayupress
3. Identify most recent valid backup: ls -lt /data/backups/*.db.gz | head -5
4. Restore from backup (see RB-07)
5. File incident report in docs/incidents/

RB-04: TLS Certificate Expired

1. Check expiry: certbot certificates
2. Force renewal: certbot renew --force-renewal
3. Test nginx config: nginx -t
4. Reload nginx: systemctl reload nginx
5. If ACME challenge fails, check firewall: ufw status (port 80 must be open for HTTP-01)

RB-05: High Error Rate (>1% 5xx)

1. Check error logs: journalctl -u vayupress | grep 'level=error' | tail -20
2. Check /health/dependencies for failing deps
3. If database errors: check WAL size, run integrity check (RB-03 step 1)
4. If Meilisearch errors: systemctl restart meilisearch (app degrades gracefully)
5. If nginx errors: nginx -t && systemctl reload nginx

RB-06: Disk Space Running Low (<20% free)

1. Check usage: df -h /data
2. Find large files: du -sh /data/* | sort -rh | head -10
3. Rotate logs: logrotate -f /etc/logrotate.d/vayupress
4. Clean orphaned media: curl -X POST http://localhost:8080/api/v1/admin/cleanup-orphans -H "Authorization: Bearer $VAYU_API_KEY"
5. Archive old backups: keep last 30 days, compress older
6. If WAL is large: checkpoint (RB-02 step 4)

RB-07: Restore from Backup

# Stop application
systemctl stop vayupress

# List available backups
ls -lt /data/backups/

# Restore (replace TIMESTAMP with target backup)
cp /data/vayupress.db /data/vayupress.db.pre-restore
zcat /data/backups/vayupress-TIMESTAMP.db.gz > /data/vayupress.db

# Verify restored DB
sqlite3 /data/vayupress.db "PRAGMA integrity_check;"

# Set correct ownership
chown vayupress:vayupress /data/vayupress.db

# Start application
systemctl start vayupress

# Verify health
curl http://localhost:8080/health

RB-08: Fail2ban Blocking Legitimate Users

1. Check if IP is banned: fail2ban-client status nginx-limit-req
2. Unban specific IP: fail2ban-client set nginx-limit-req unbanip <IP>
3. Whitelist if needed: add to /etc/fail2ban/jail.local under [DEFAULT] ignoreip
4. Reload: fail2ban-client reload

RB-09: API Key Rotation

# Generate new key
NEW_KEY=$(openssl rand -hex 32)
echo "New API key: $NEW_KEY"

# Update environment
# Edit /etc/vayupress/vayupress.env
# Change VAYU_API_KEY=<new_value>

# Restart to apply
systemctl restart vayupress

# Update all API clients with new key
# Verify health
curl -H "Authorization: Bearer $NEW_KEY" http://localhost:8080/health

Monitoring Metrics (Prometheus)

Key metrics exposed at http://localhost:8080/metrics (internal only):

Scheduled Operations

Capacity Planning

Incident Classification

Security incidents: email security@vayupress.com immediately.

Emergency Contacts