Self-hosting the Aetherin server
The complete path from an empty host to a running, observable, backed-up deployment. Every command runs from aetherin-deploy/.
- Time: ~30 minutes, most of it waiting for images and certificates.
- What you need: a host with Docker + Compose, a domain with DNS pointing at
it, and an Infisical account (or a local .env with the same variables).
0. Prerequisites
docker --version # 24+ with the compose plugin
docker compose version
dig +short A <your-domain> # resolves to the host?Ports 80 and 443 must be reachable from the internet (Caddy terminates TLS). Everything else runs on an internal Docker network.
1. Get the code and the environment
git clone git@github.com:AetherinLLM/aetherin-deploy.git
cd aetherin-deploy
cp .env.production.example .envFill in .env:
| Variable | How to get it |
|---|---|
API_DOMAIN | your domain, e.g. api.example.com (A/AAAA → the host) |
POSTGRES_PASSWORD, REDIS_PASSWORD | generate: openssl rand -base64 24 |
ROOT_KEY | openssl rand -base64 32 — back this up offline, losing it loses every encrypted credential |
CORE_SERVICE_TOKEN | openssl rand -hex 32 — shared with the middleware only |
BOOTSTRAP_ADMIN_EMAIL / BOOTSTRAP_ADMIN_PASSWORD | the first operator account |
CORS_ORIGINS | your frontend origin(s), comma separated |
STRIPE_PAYMENT_METHODS | only methods your Stripe account accepts (see SECRETS.md) |
make bootstrap generates ROOT_KEY and CORE_SERVICE_TOKEN for you if they are empty.
Secrets from Infisical instead
If you keep secrets in Infisical (recommended for anything but a laptop):
export INFISICAL_DOMAIN=https://eu.infisical.com
make import-env # -> .env.infisical (mode 600)
make up # compose layers .env.infisical over .envcheck-secrets.sh verifies that the expected names exist before you start (make check-secrets). The Stripe key is deliberately prod-only — a staging key can create real charges.
2. Start the stack
make up
make logsExpected order: postgres → redis → migrate → core → caddy (plus prometheus, grafana and the exporters).
Verify:
curl -fsS "https://$API_DOMAIN/v1/healthz" # liveness
curl -fsS "https://$API_DOMAIN/v1/readyz" # readiness (db + redis)
curl -fsS "https://$API_DOMAIN/v1/version" # build info/v1/readyz must be ok before you point a client at it.
3. First login
The first operator is created automatically on an empty database (from BOOTSTRAP_ADMIN_EMAIL/BOOTSTRAP_ADMIN_PASSWORD, or ADMIN_TOKEN from Infisical when no explicit password is set).
curl -sS -c cookies.txt -X POST "https://$API_DOMAIN/v1/auth/login" \
-H 'content-type: application/json' \
-d '{"email":"admin@example.com","password":"…"}'Then create a client key and make a request — see the API quickstart.
4. TLS
Caddy obtains and renews certificates automatically for API_DOMAIN. It needs DNS to resolve to the host and ports 80/443 open. docker compose logs caddy shows the ACME result; a failed challenge usually means the port is blocked or the DNS record is wrong.
The middleware adds HSTS and the other hardening headers itself — you do not need to configure them at Caddy.
5. Observability
| Surface | Where |
|---|---|
| Prometheus | http://<host>:9090 (targets + rules) |
| Grafana | http://<host>:3000 (dashboard *Aetherin Core*) |
| Alerts | alerts.yml (26 live rules) + alerts.pending.yml (rules waiting for metrics that do not exist yet) |
| Incidents | `docs/incident-response.md` (./incident-response.md) |
| Health | /v1/healthz, /v1/readyz |
Known gap: there is no Alertmanager, so nothing pages you. Until one is wired, someone has to look at Prometheus → Alerts.
6. Backups and restore
make backup # dump Postgres to $BACKUP_DIR
make drill # restore the newest dump into a throwaway DB and verify it
make drill DRILL_ARGS=--fixture # no backup needed: use a generated fixture
make restore DUMP=/var/backups/aetherin/pg-aetherin-….sql.gz CLEAN=1make drill restores into an isolated container with its own volume and runs integrity checks (schema version, required tables, foreign keys, row counts). A red drill means the restore path is broken — fix it before you need it.
Install the timers on the host:
sudo cp systemd/aetherin-backup.{service,timer} /etc/systemd/system/
sudo cp systemd/aetherin-drill.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now aetherin-backup.timer aetherin-drill.timer
systemctl list-timers | grep aetherinRTO/RPO, retention and offsite copies: see the *Disaster recovery* section of `README.md`.
7. Upgrades
git pull
make check-secrets # fail early if a secret is missing
make up # recreates changed services
make migrate # migrations run automatically on start
curl -fsS "https://$API_DOMAIN/v1/readyz"Migrations are forward-only (golang-migrate); take a backup before a release that changes the schema. The migrate service runs on every start and blocks core until it succeeds.
8. Operating notes
- The core is not exposed publicly — only Caddy is. Postgres, Redis and the
exporters are on the internal network.
- `ROOT_KEY` is the crown jewel. It decrypts the sealed provider
credentials. Back it up offline; losing it makes a restored database unusable for every provider.
- Secrets in Infisical are read at process start. Changing one means a
restart (make up) — and, for provider keys, the core re-seals them on boot.
- Logs do not contain prompts or completions. If you enable body logging
for debugging, the privacy notice in aetherin-gtm/legal/datenschutz.md becomes wrong — update it in the same change.
Troubleshooting
| Symptom | Likely cause | Check |
|---|---|---|
/v1/readyz stays not ready | Postgres or Redis not healthy | docker compose ps, make logs |
| Core restarts | migration failed | docker compose logs migrate |
401 on every request | wrong CORE_SERVICE_TOKEN between core and middleware | compare the value in both places |
Provider health down | wrong key or wrong base URL | /v1/providers → health check |
402 for a client | that key's budget is spent | /v1/usage, raise the budget |
503 secrets_unavailable | ROOT_KEY missing | grep ROOT_KEY .env |
| Caddy cannot get a certificate | DNS/ports 80-443 | docker compose logs caddy |