Alerts
The seven rules, what opens and resolves each one, and how deduplication keeps the list worth reading.
The rules
| Kind | Severity | Opens when | Threshold |
|---|---|---|---|
DISK_PRESSURE | WARNING, CRITICAL at ≥ 95% | Disk usage crosses the threshold | ALERT_DISK_USED_FRACTION (0.85) |
MEMORY_PRESSURE | WARNING | Memory stays above the threshold for the sustained window | ALERT_MEMORY_USED_FRACTION (0.9), ALERT_MEMORY_SUSTAINED_MINUTES (5) |
CONTAINER_RESTART_LOOP | CRITICAL | A container restarts more than n times in the window | ALERT_RESTART_COUNT (3), ALERT_RESTART_WINDOW_MINUTES (5) |
HEALTH_CHECK_FAILING | CRITICAL | An application reports unhealthy | — |
CERTIFICATE_EXPIRING | WARNING, CRITICAL at ≤ 3 days | A certificate is near expiry | ALERT_CERTIFICATE_EXPIRY_DAYS (14) |
DEPLOYMENT_FAILED | WARNING | A deployment fails | — |
BACKUP_FAILED | CRITICAL | A managed-service backup fails | — |
The thresholds are configurable because a box running one application and a box running twenty disagree about what "too full" means.
When rules are evaluated
- Sampled rules — disk, memory, restart loops, health checks — against each metrics tick, every 15 seconds.
- Standing rules — certificate expiry — on the maintenance cron.
- Event rules — failed deployments and failed backups — when the event happens.
Deduplication
Every alert carries a fingerprint, and an open alert with the same fingerprint is updated
rather than duplicated. lastSeenAt moves; a second row does not appear.
That is enforced by a partial unique index on open alerts, written by hand in the migration because Prisma cannot express one.
Every rule both opens and resolves. An alert list that only grows is a list nobody reads, and the whole value of the feature is that an open alert means something is wrong right now.
Status
OPEN or RESOLVED. POST /alerts/:id/resolve closes one by hand.
GET /alerts/summary returns three counts — open, critical, warning — for the shell's badge,
so the badge needs no page of rows to render.
Delivery
Alerts open regardless of delivery. If nobody is receiving them, check the portal first to confirm the condition was noticed at all; if it was, the delivery side is at fault. See notification channels.
Delivery runs through the notify queue, one job per channel, so a retry re-attempts only the
receiver that failed.
| Setting | Default |
|---|---|
NOTIFY_MAX_ATTEMPTS | 4 |
NOTIFY_BACKOFF_MS | 10000 |
NOTIFY_CONCURRENCY | 4 |
NOTIFY_WEBHOOK_TIMEOUT_MS | 10000 |