Incident response runbook
Operational runbook for the Aetherin self-hosted stack: how a page becomes an incident, who decides, what to do in the first 15 minutes, and what to write down afterwards.
- Alert rules → `alerts.yml` (../alerts.yml) (live) and
`alerts.pending.yml` (../alerts.pending.yml) (metric contract, not loaded)
- Backups, restore steps and measured RTO/RPO → Disaster recovery: RTO / RPO (../README.md#disaster-recovery-rto--rpo)
- Secrets and rotation → `secrets-runbook.md` (./secrets-runbook.md)
- Validate a rules change before shipping it →
make check-alerts
Hard rule: in an incident, stop the bleeding before you understand it.
Roll back or restart, get the service serving, *then* investigate with the
evidence you preserved.
1. Detection & delivery
Signals, in order of trustworthiness:
- Prometheus alerts (
alerts.yml) — evaluated every 15s, visible at
http://<host>:9090/alerts. Rule metadata (severity, summary, first action, runbook link) is attached to every alert.
- Grafana (
http://<host>:3000, dashboard *Aetherin Core*). - External probe — a request to
https://$API_DOMAIN/healthzfrom outside
the host (uptime checker, or curl from your laptop). This is the only way to see "the host is up but the internet cannot reach it".
- User reports — treat a credible report as SEV3 or worse until disproved.
Known gap — no paging. This stack has no Alertmanager and no receiver, so alerts are *evaluated* but nobody is notified automatically. Until that is wired (Alertmanager service + webhook/Slack/email receiver, referenced in prometheus.yml), someone must watch Prometheus → Alerts, and the on-call rotation below is a *best-effort sweep* rather than a pager rotation. See prometheus.yml for the commented alerting: block.
Capture this before you touch anything in an outage you intend to postmortem:
date -u; docker compose ps; docker version --format '{{.Server.Version}}'
docker compose logs --since=30m core postgres redis caddy > /tmp/incident-$(date -u +%Y%m%dT%H%M%SZ).log
curl -fsS http://127.0.0.1:9090/api/v1/alerts | jq '.data.alerts[] | {name:.labels.alertname,state:.state}'
cp -a reports/drill-latest.json /tmp/ 2>/dev/null || true2. Severity matrix
| Severity | Definition | Examples | Response time | Comms cadence | Who decides |
|---|---|---|---|---|---|
| SEV1 | Total loss of the customer-facing API or of data integrity. | Core down (AetherinCoreDown); Postgres down/refusing writes (AetherinPostgresDown); public endpoint unreachable; data corruption or suspected data loss; secret/key compromise. | Page immediately, start within 15 min, 24/7. | Status page + stakeholder update every 30 min until mitigated, then hourly. | Incident commander (on-call engineer, unless the founder/CTO takes over). |
| SEV2 | Major degradation: a large share of requests fail or the stack is one failure away from SEV1. | Redis down (AetherinRedisDown); 5xx > 1% (AetherinProxy5xxRate); traffic collapse (AetherinProxyTrafficCollapse); disk < 5% (AetherinDiskSpaceCritical); connections > 95% (AetherinPostgresConnectionsExhausted); backup stale > 36 h (AetherinBackupStale). | Start within 30 min, business hours commitment relaxed for on-call. | Update every 2 h; status page if customer-visible > 30 min. | On-call engineer; escalate to owner if not mitigated in 2 h. |
| SEV3 | Minor/partial degradation with a workaround, or a reliability risk that must be fixed this week. | Connections > 85%, p95 latency > 10 s, backup too small (AetherinBackupTooSmall), drill failing/stale, deadlocks, rollback spike, disk < 15 %, host memory low. | Acknowledge same day, fix within 3 business days. | Ticket + weekly ops note. | On-call engineer / service owner. |
| SEV4 | Cosmetic, cost, or hygiene issue. No user impact. | Evictions, exporter down (blind spot, not an outage), Grafana/dashboard issues, config-reload failures, certificate expiring in > 14 days. | Next planning cycle. | Overdue backlog review. | Service owner. |
Rules of thumb:
- When in doubt, go up a severity. Downgrading later is cheap, apologising
late is not.
- Any suspected data loss or secret exposure is SEV1, even if the service
looks healthy.
- The incident commander decides severity and comms; anyone may declare an
incident and hand it over.
- Record the *severity, start time, and commander* in the incident channel in
the first message — before analysis.
3. On-call rotation expectations
This is a small-team rotation; the commitments are deliberately modest but explicit:
- Coverage: one primary on-call per week, with one named secondary. Handover
happens Monday 10:00 local, in writing (what is flapping, what is open).
- Response: SEV1 acknowledged within 15 minutes, 24/7. SEV2 within 30
minutes during waking hours. SEV3/SEV4 next business day.
- Tools: SSH access to the host, GitHub access, Infisical access for the
environment, and a working local docker + make (the drill needs it). Verify each of these during handover — an incident is the wrong time to discover a missing credential.
- Handover checklist:
make check-secrets,/alertsreviewed (nothing
firing/flapping), systemctl list-timers aetherin-* shows backup *and* drill scheduled, newest reports/drill-*.json is PASS, disk on the host > 25 %.
- No pager yet: because delivery is not configured, the primary is expected
to check /alerts and /healthz at least twice a day, and to treat user reports as the real pager.
- Escalation: on-call → service owner → founder. Escalate immediately for
data loss, suspected compromise, or anything customer-visible over 30 minutes.
- Fatigue rules: if you have been paged more than twice in a night, hand over
rather than continue; a flapping alert must be fixed or silenced with an expiry, never muted silently.
- After any SEV1/SEV2: postmortem within 5 business days (template in §7),
blameless, with at least one concrete action item and an owner.
4. First 15 minutes checklist
Work top to bottom; do not skip step 1.
- Declare. Create the incident channel, set severity, name yourself
commander, post: *what is broken, since when, who is affected*.
- Stop the bleeding (60 s decision). If the incident started within ~2 h of
a deploy or config change, roll back first (§5) — do not debug forward.
- Quantify impact. `curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n'
https://$API_DOMAIN/healthz` from *outside* the host, plus a real API call. Note the exact failure mode (5xx / timeout / TLS / DNS).
- Triage the stack (in this order — each is a 30-second check):
docker compose ps # anything exited / unhealthy / restarting?
docker compose logs --tail=100 core # panics, DB errors, OOM?
docker compose logs --tail=50 postgres redis caddy
df -h / # disk full is the most common silent killer
curl -fsS localhost:9090/-/ready && curl -fsS localhost:9090/-/healthy- Preserve evidence (§1 command block) *before* restarting containers, if
you can do it in under a minute. docker compose logs on a restart is gone.
- Apply the matching runbook (§6) and say what you are doing in the channel
before you do it.
- Restore service, then verify with a real request (not just
/healthz):
authenticate and make one API call.
- Set the comms clock. SEV1: next update in 30 minutes, even if the update
is "still investigating".
- Hand off explicitly if you need sleep or help — name your successor in the
channel; never just stop posting.
5. Rollback and restore
Rollback (preferred for regression after a deploy). The core is an image tag, so rollback is a one-line change — no database rollback is attempted:
# 1. pin the previous known-good tag in .env
# CORE_TAG=<previous-good-sha-or-tag>
docker compose up -d core
docker compose logs -f --tail=50 core
curl -fsS https://$API_DOMAIN/healthzRollback is safe because migrations are forward-only and additive; a *newer* schema with an *older* binary is the supported direction. If a migration itself is the problem, you are in restore territory, not rollback.
Restore (data loss / corruption / dead host). Full procedure, retention and measured timings live in Disaster recovery: RTO / RPO (../README.md#disaster-recovery-rto--rpo). Short form:
make up # throwaway/new host stack
make restore DUMP=/var/backups/aetherin/pg-aetherin-<ts>.sql.gz CLEAN=1
make migrate # if the dump predates head
curl -fsS https://$API_DOMAIN/readyzTwo hard prerequisites that are *not* in the database dump:
- `ROOT_KEY` — envelope-encryption root key. Without it, restored provider
credentials are undecryptable. Keep it in the secure vault and test the restore with it, not just the SQL.
- `.env` — DB/Redis passwords, tokens, domains.
Never restore onto a live database without a dump taken *after* deciding to restore. If the live volume may still hold better data than the backup, snapshot it first (copy the pgdata volume, or make backup) — do not overwrite the only copy of the freshest data.
6. Alert catalogue (what fires, what you do)
Each entry is the runbook target of the runbook_url in alerts.yml. Metric sources are listed so you can confirm the alert is not firing on stale data.
AetherinCoreDown
up{job="aetherin-core"} == 0 for 2m · critical · SEV1. docker compose ps core; docker compose logs --tail=100 core; is postgres healthy (AetherinPostgresDown)? If a deploy preceded it, roll back (§5). If the container is healthcheck-failing, test curl localhost:8080/healthz from inside the network (docker compose exec core wget -qO- http://127.0.0.1:8080/healthz).
AetherinCoreScrapeMissing
absent(up{job="aetherin-core"}) for 5m · critical (observability) · SEV1. up has no series at all, so availability is unmonitored. Check the job/target in prometheus.yml, then docker compose restart prometheus and verify Targets. Treat as SEV1 because you are blind.
AetherinExporterDown
up{job=~"...-exporter"} == 0 for 5m · warning · SEV4. A blind spot, not an outage: restart the exporter and check its credentials (POSTGRES_PASSWORD, REDIS_PASSWORD) — a rotated password breaks the exporter before it breaks the app.
AetherinPrometheusConfigReloadFailed
prometheus_config_last_reload_successful == 0 for 10m · warning · SEV4. make check-alerts; fix YAML; curl -X POST localhost:9090/-/reload.
AetherinPrometheusRuleEvalFailing
increase(prometheus_rule_evaluation_failures_total[15m]) > 0 · warning · SEV3. Find the erroring rule in Prometheus → Alerts. Usually a typo or a metric that was renamed upstream.
AetherinPostgresDown
pg_up == 0 for 2m · critical · SEV1. docker compose ps postgres; logs; df -h / (a full disk stops Postgres first); docker compose restart postgres once. If it will not start on the data directory, stop and go to restore (see Disaster recovery (../README.md#disaster-recovery-rto--rpo)) — do not run initdb/pg_resetwal on production data.
AetherinPostgresConnectionsHigh / AetherinPostgresConnectionsExhausted
sum(pg_stat_activity_count) / max(pg_settings_max_connections) > 0.85 (warning) / > 0.95 (critical).
SELECT state, usename, application_name, count(*) FROM pg_stat_activity GROUP BY 1,2,3 ORDER BY 4 DESC;
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '15 minutes';If it is a leak, restart core; if it is load, raise max_connections (memory cost per connection!) and plan pooling.
AetherinPostgresDeadlocks
increase(pg_stat_database_deadlocks[10m]) > 0 · warning · SEV3. Collect the deadlock detected log lines, identify the two statements, and file a bug with both — this is a code-shape problem, not an ops knob.
AetherinPostgresRollbackSpike
10 % of transactions rolling back for 10m · warning · SEV3.
Correlate with the deploy timeline; check for constraint/serialization errors and connection saturation. Roll back if a deploy lines up.
AetherinRedisDown
redis_up == 0 for 2m · critical · SEV2. Rate limiting and cache disappear; core should still serve. Confirm REDIS_PASSWORD matches .env, restart once. Do not restore Redis data — it is disposable, and a restore adds risk without fixing availability.
AetherinRedisMemoryHigh / AetherinRedisEvictions
Memory > 90 % of maxmemory (warning) or any evictions in 15m (warning) · SEV4. Rate-limit counters can be evicted silently. Inspect big keys, add TTLs, raise maxmemory only with host headroom.
AetherinDiskSpaceLow / AetherinDiskSpaceCritical / AetherinDiskWillFillIn24h
Filesystem < 15 % (warning/SEV3), < 5 % (critical/SEV2), or projected to hit zero within 24h (warning/SEV3).
df -h /; du -sh /var/lib/docker/volumes/* | sort -h | tail
docker image prune -f; docker system df
du -sh /var/backups/aetherin; ls -lt /var/backups/aetherin | headFree space, but never delete pgdata, the newest backup, or ROOT_KEY material. If Postgres already stopped with a full disk, free space first, then verify it starts and check for WAL corruption before trusting it.
AetherinHostMemoryLow
< 10 % available memory for 15m · warning · SEV3. docker stats --no-stream; find the leaker; restart it before the OOM killer picks Postgres.
AetherinBackupStale
No successful dump in 36 h · critical · SEV2. RPO is documented as ≤ 24 h, so 36 h is already a breach. Run make backup immediately, read the failure from journalctl -u aetherin-backup.service -n 100 (or the backup service logs), verify the new file with make drill, and open SEV2 until a fresh dump is verified.
AetherinBackupMetricMissing
absent(aetherin_backup_last_success_timestamp_seconds) for 6h · warning · SEV4. Either no backup ever ran or monitoring/textfile is not mounted into node-exporter. Check ls -l monitoring/textfile/*.prom (must be world-readable) and make backup.
AetherinBackupTooSmall
Newest dump < 1 KiB for 10m · warning · SEV3. Do not trust that file. Take a manual backup, then verify the *previous* dump with make drill DRILL_ARGS='--backup /var/backups/aetherin/pg-aetherin-<ts>.sql.gz'. Look for "disk full" in the backup logs.
AetherinDrillStale
No passing drill in 35 days · warning · SEV3. make drill; read reports/drill-latest.json; check systemctl status aetherin-drill.timer (Persistent=true should have caught up after downtime). Fires on a fresh host where no drill has ever passed.
AetherinDrillFailing
aetherin_drill_last_run_success == 0 for 1h · warning · SEV3. This is the loudest DR signal there is: a restore may not work. Read the checks array in reports/drill-latest.json, journalctl -u aetherin-drill.service, fix the failing check, re-run make drill.
AetherinProxy5xxRate / AetherinProxyErrorRate
5xx > 1 % (critical/SEV2) / handled proxy errors > 2 % (warning/SEV3) for 10m — metrics from aetherin-middleware (proxy_requests_total, proxy_errors_total). Dormant until a target is added to monitoring/targets/middleware.json. Break down by code/status; check AetherinCoreDown and upstream provider status; roll back the middleware/core tag if a deploy lines up.
AetherinProxyLatencyP95High
p95 > 10 s for 10m (proxy timeout is 30 s) · warning · SEV3. Separate provider latency from core/DB latency before touching any timeout.
AetherinProxyTrafficCollapse
Request rate ≈ 0 while healthy an hour ago · critical · SEV2. Suspect Caddy, DNS for $API_DOMAIN, certificate failure, or a routing change — not a quiet period. Check docker compose logs caddy, then resolve the domain from outside.
7. Postmortem template
# Postmortem — <title> (<SEV>) — <YYYY-MM-DD>
- Incident commander:
- Participants:
- Detection: <alert / user report / probe> at <time>
- Declared: <time> (severity) Mitigated: <time> Resolved: <time>
- **Time to detect (TTD):** **Time to mitigate (TTM):** **Time to resolve (TTR):**
- User impact: <who, how many, how long, what failed>
## What happened (timeline, UTC)
| time | event |
|---|---|
| | |
## Root cause
<the mechanism, not the person>
## What went well / what went badly
-
## Where we got lucky
-
## Action items
| action | owner | due | ticket |
|---|---|---|---|
| | | | |
## Detection changes
<new alert / threshold / probe — or "none needed because …">
## Data or secret loss?
<yes/no + evidence; if yes, link the restore/drill report>Rules: blameless; one owner and one due date per action; every SEV1/SEV2 yields at least one change to detection or to the runbooks; if a runbook step was wrong, fix the runbook in the same PR class as the action items.
8. Never do this
- Never
docker compose down -von production.-vdeletespgdata. The
drill script is the only thing allowed to use -v, and only against its own throwaway project.
- Never run a restore against the production database without a fresh backup
of the *current* data, and never with --force on a live database you have not snapshotted.
- Never delete the newest dump,
pgdata, theROOT_KEY, or.envto free
disk space. Free space somewhere else.
- Never
pg_resetwal,initdb, or hand-edit the data directory to "fix" a
broker Postgres — restore instead.
- Never rotate the DB/Redis password in one place (Infisical,
.env,
exporter) without doing the others; the exporter will silently go blind.
- Never silence or delete a firing alert to make a dashboard green. Silence
with an expiry and a reason, or fix it.
- Never commit
.env,ROOT_KEY, dump files, orreports/(already
gitignored — keep it that way).
- Never "test" the restore path by hand-waving: if the restore is untested,
you do not have a backup, you have a file.
- Never debug forward during an incident when a rollback takes one command.
- Never let a SEV1 run without a named commander and a comms clock.
9. Known detection gaps
These are real, current blind spots. They are *documented*, not papered over with rules on metrics that do not exist (alerts.pending.yml holds the ready-to-use rules for when the metrics land):
| Gap | Impact | What would close it |
|---|---|---|
| Core 5xx rate, p95 latency, request-rate collapse | A core regression that returns errors without going down is invisible. | A labelled counter/histogram in core (http_requests_total{status} + http_request_duration_seconds_bucket). Today /metrics emits only a constant aetherin_requests_total that is never incremented. |
Core readiness (/readyz) | up only proves /metrics answers. | blackbox_exporter HTTP probe on /readyz. |
| Public endpoint reachability from the internet | "host up, world can't reach it" (DNS/TLS/Caddy) looks healthy from inside. | blackbox_exporter HTTP probe on https://$API_DOMAIN/healthz. |
| TLS certificate expiry | Silent expiry if Caddy's automatic renewal breaks. | blackbox_exporter TLS probe (probe_ssl_earliest_cert_expiry). |
| Offsite/independent backup copy | Host loss = backup loss (they share the failure domain). | Encrypted upload to object storage (rclone/restic) + an upload-age metric. |
| Backup encryption | Dumps are gzip'd plain SQL, readable by anyone with the file. | Encrypt before writing (age/GPG) or use pgBackRest/restic with encryption. |
| Redis not covered by restores | Acceptable today (cache only) — re-evaluate if queues become authoritative. | Promote Redis to backed-up state or document the loss explicitly. |
| Grafana dashboard queries | grafana/dashboards/aetherin-core.json queries http_requests_total, which core does not expose, so panels are empty. | Point panels at metrics that exist (see the same core instrumentation gap). |
| Alert delivery | Alerts are visible but never pushed anywhere. | Alertmanager + a receiver, wired in prometheus.yml. |
| Middleware proxy metrics | proxy_* rules stay dormant with no scrape target. | Add the middleware metrics URL to monitoring/targets/middleware.json. |
If you close one of these, move the corresponding rule out of alerts.pending.yml into alerts.yml, run make check-alerts, and update this table in the same change.