Operations
Routine checks
Where everything lives on the box, and the commands worth running when you want to know it is fine.
For whoever is holding the pager. Every procedure here assumes root on the box and nothing else.
Where everything is
| Thing | Path |
|---|---|
| Generated credentials | /etc/sthora/.env (0600) |
| Master encryption key | /etc/sthora/master.key (0600) |
| Platform compose file | /etc/sthora/docker-compose.yml |
| Compose image and hostname variables | /etc/sthora/compose.env |
Source trees, build logs, backups, known_hosts | /var/lib/sthora/ |
| Traefik certificates | the acme.json volume of the sthora-proxy project |
Checks
| Check | Command |
|---|---|
| Platform health | docker exec sthora-api curl -fsS localhost:4600/health |
| What is running | docker compose -p sthora ps |
| API logs | docker compose -p sthora logs -f api |
| Worker logs | docker compose -p sthora logs -f worker |
| Disk | df -h / and docker system df |
| Open alerts | the portal's alerts page, or GET /alerts?status=OPEN |
| Agent state | the portal's server page — version, last seen, and whether a socket is open right now |
| Agent logs, on the target machine | docker logs -f sthora-agent |
What runs on its own
| Job | When |
|---|---|
| Metric sampling | Every 15 seconds |
| Metric rollups and retention | Maintenance cron |
| Dangling image prune, build cache prune | Nightly |
| Certificate expiry check | Maintenance cron |
| Platform database dump | Nightly, to /var/lib/sthora/backups/platform/ |
| Managed-service backups | Whatever cron each service is set to |
Reclaiming disk
The pipeline refuses to start a build below DEPLOY_MIN_FREE_DISK_FRACTION (15% free), and
that refusal is the message you will see. Reclaim in this order:
docker image prune --force # dangling images
docker builder prune --force # build cache, usually the largestBoth run nightly on their own. If they are not enough, lower imageRetentionCount on the
noisiest applications — each retained version is a full image.
Never run docker system prune -a
It removes every image no running container uses, which includes the previous versions rollback depends on.
Security hygiene
- The firewall allows inbound 22, 80, and 443 only. Docker's published ports bypass ufw's
chains, so never start a container by hand with
-pon this box. - Turn on two-factor authentication for every account.
- Use API tokens for CI rather than sharing a password.
- The audit log records every mutating action, including reads of a managed service's connection details.