Troubleshooting
The incidents that actually happen, in the order to work through them.
A deployment is stuck at DEPLOYING
The worker was probably killed. Restarting it reaps anything left claiming to run:
docker compose -p sthora restart workerThe boot-time reaper marks interrupted deployments FAILED with a message saying the worker
restarted.
Deploy again once you understand the cause. A build that was OOM-killed will be OOM-killed again.
A server shows "not connected"
Nothing will deploy to it until an agent is back. Work outwards from the machine:
# On the target machine
docker ps --filter name=sthora-agent
docker logs --tail 50 sthora-agent| What the log says | What it means |
|---|---|
Refused by the control plane … Not retrying | The token was rotated or the server was removed. Rotate it in the portal and re-run the command with the new one. |
Disconnected (…). Reconnecting in Ns | It is trying. The backoff caps at a minute; if the control plane is up and reachable, look at TLS and at whatever proxy sits in front of the portal. |
This control plane speaks agent protocol N | The agent is older than the control plane. Pull the image again and recreate the container. |
| Nothing, and no container | It was never started, or the machine rebooted without --restart unless-stopped. |
Applications already running on that server keep running.
Builds are failing on disk
The pipeline refuses to start below the free-disk floor, and that refusal is the message you will see.
docker image prune --force # dangling images
docker builder prune --force # build cache, usually the largestIf they are not enough, lower imageRetentionCount on the noisiest applications. Never run
docker system prune -a — see routine checks.
An application is unreachable but its containers are up
Check in this order:
DNS. The hostname must resolve to this box. The portal shows what it last resolved to; a hostname that stops resolving here is un-verified on the next check and its router is dropped, by design.
The certificate. The portal shows the status read from Traefik's acme.json, never
assumed. PENDING for more than a few minutes means issuance is failing — check
docker compose -p sthora-proxy logs traefik.
The router. docker inspect <container> | grep traefik should show the labels. Every
traefik.* label a user's own compose file declared is stripped, so if the labels are missing
the hostname is not verified.
Let's Encrypt is refusing to issue
Production allows five failures per hostname per hour. The usual causes are port 80 not reaching the proxy — the HTTP-01 challenge needs it even for a certificate served on 443 — and a contact address the CA rejects.
Switch to staging while debugging:
sed -i 's/PROXY_ACME_STAGING=false/PROXY_ACME_STAGING=true/' /etc/sthora/.env
docker compose -p sthora restart api workerStaging issues untrusted certificates from a directory with far looser limits. Remember to switch back.
A managed service will not start
docker compose -p st-svc-<slug> logsThe most common cause on a first start is a volume that already holds data from a different major version. Postgres refuses to start against an on-disk format it did not write, and the log says so explicitly.
There is no in-place fix: restore a backup into a new service.
Nobody is receiving alerts
Alerts open regardless of delivery — check the portal first to confirm the condition was noticed at all. If it was, the delivery side is at fault:
- The channel's last error is shown on the notification-channels page. That is the receiver's own message.
- Email is skipped entirely when no transport is configured, and says so in the API log rather than failing.
- Delivery runs through the
notifyqueue. If Redis is down, alerts are raised and stored but nothing is sent.
The dashboard's numbers describe a container, not the machine
HOST_PROC_PATH, HOST_SYS_PATH, and HOST_ROOT_PATH are unset or not bind-mounted. See
metrics. The dashboard shows a
banner and the API logs a warning at boot.
A clone started failing with a host-key error
Host keys are trusted on first use and pinned afterwards. A Git host whose key changed makes the next clone fail loudly rather than quietly trusting a substituted key.
If you rebuilt that host on purpose, remove its line from /var/lib/sthora/known_hosts.
Deployments queue and never start
Check Redis. BullMQ stores job state there, and maxmemory-policy must be noeviction — an
LRU policy silently evicts job state and deployments disappear with no error.
source /etc/sthora/.env
docker exec sthora-redis \
redis-cli -a "$REDIS_PASSWORD" --no-auth-warning config get maxmemory-policyThe platform's own Redis is started with --maxmemory-policy noeviction by its compose file,
so an answer other than noeviction means something changed it.