Alerts and silences
Read what an alert found, run its fix and silence planned work.
Alerts
seawall alerts # what fires
seawall alerts ContainerDown # one alert with what it found and what fixes it
seawall silence add nextcloud --for 2h --comment "upgrade" --yes
seawall silence alert DiskHigh --label mountpoint=/ --for 1h --yes
seawall silence ls
seawall silence rm <id> --yes
seawall reboot-prep --yes # 15 minutes for a planned reboot
alerts <name> reads the alert's cause live. Found lists what it saw. Do lists the commands that fix it, each a dry run first unless it acts at once. In the command centre each Do line is a number key, marked now where it acts at once. Under CloudflareOriginPullsCertExpiring, proxy aop renew and, for a host set up by hand, proxy aop HOST --guided act at once and have no mark.
init writes a rule file for each pack. The host, containers, security, updates and backups packs come with the monitoring stack. The logs pack reads Loki and comes with the logging step. Its sev0 SSH login alert needs [logging] ssh_from. The audit step adds rules-seawall-audit.yaml, which holds AuditPipelineStale. AdminLoginBruteForce and SiteErrors read the proxy's logs, so they exist only when Nginx Proxy Manager is the proxy.
Daily findings (SecurityUpdateCritical, SeawallImageCritical) carry the label page=daily. Where seawall routes the alerts, it pages them once a day within notifications.page_hours (08:00-23:59 by default). A finding that fires inside the window pages at once, and one that fires outside it pages when the window opens. Where you route alerts yourself, give page=daily a route placed before your severity=sev1 route, with active_time_intervals set to [seawall-page-hours] and repeat_interval set to 23h55m. The first matching route wins, and both daily rules are sev1.
A silence never covers a sev0 alert. silence add refuses a name that is neither a container nor a stack. silence alert takes any alert name and labels, so check the name with seawall alerts <name> first. A silence lasts from 1 minute to 30 days.
Where alerts go
The Pages and Rollup rows in seawall config set where alerts go: Discord, Slack, Telegram, Pushover, ntfy, Gotify, Matrix, a webhook or email. Pages takes sev0 and sev1 at once and daily findings within notifications.page_hours. Grafana's DatasourceError and DatasourceNoData, and any alert with no sev label, go to Pages too. Rollup takes sev2 and sev3 in the 09:00-09:15 window on the host's clock. With no Rollup row set, they go to Pages. Each secret is typed hidden and kept where only Grafana reads it. A row can send a test or be removed. seawall notify test sends one to Pages, Rollup or both through Grafana, which holds the secrets. seawall never reads or prints them.
Email goes through the host's one mail server, set in Grafana's own settings with its password in a file. Grafana must use STARTTLS, so a server that doesn't offer it fails and the password never crosses in clear. notifications.smtp.plain allows clear text when the server offers no TLS. Port 465 is TLS from the start.
Rule packs
monitoring.packs picks the rule packs seawall's monitoring keeps, from host, containers, security, updates, backups and logs. The old names health, audit and probes read as the packs they became. A pack turned off keeps its sev0 rules. One that holds sev1 rules names them and goes off only with the pack's name typed. A pack rule whose title the host already has in a rule file of its own is left out, so nothing pages twice. init keeps each file to this version's. One you edited is left as it is when seawall has a record of its last copy, and replaced, with yours kept in ~/backups/seawall/rules/, when it hasn't.
The packs hold 116 rules.
| Pack | Alerts on | Rules |
|---|---|---|
host | Disk, memory, RAID, drive wear and health, NVMe heat, failed units, the clock, seawall's facts, reboots, sites down | RootDiskCritical sev0, DiskHigh sev1, MemoryLow sev1, ScrapeTargetDown sev2, RaidDegraded sev0, DiskFillingFast sev1, TmpInodesLow sev1, RootInodesLow sev2, SystemdUnitFailed sev2, SecurityUpdatesPending sev2, NvmeHot sev2, ClockNotSynced sev2, SeawallFactsStale sev1, SeawallFactsIncomplete sev2, SeawallRebootPending sev3, HostUpgradeWorse sev1, DockerDownAfterUpgrade sev1, DriveCriticalWarning sev1, DriveMediaErrors sev1, DriveSpareLow sev1, DriveWearHigh sev2, DriveWearFast sev3, DriveUnread sev2, SiteDown sev1, SiteSlow sev3 |
containers | Containers down, unhealthy or restarting, and risky mounts and privileges | ContainerDown sev1, ContainerUnhealthy sev1, ContainerRestartLoop sev1, SeawallContainerFactsFailing sev1, ContainerNearMemoryLimit sev3, DockerSocketStale sev2, SeawallChangeFinding sev2, SeawallDeployWorse sev2, SeawallHostMount sev2, SeawallSocketMount sev2, SeawallPrivilegedContainer sev2, SeawallSecretsReadable sev2, SeawallWorldWritable sev3, SeawallRunUserWidened sev1, SeawallStackDataSetid sev2, SeawallOrphanStackData sev3 |
security | The firewall, CrowdSec, certificates, exposed ports and paths, the audit pipeline | FirewallWeakened sev0, FirewallFactsFailing sev1, UnexpectedPublishedPort sev1, UnexpectedPrivilegedContainer sev1, CrowdSecDown sev1, CrowdSecNotParsing sev1, CrowdsecHubFailing sev3, CrowdsecHubStale sev3, CrowdSecBouncerStale sev1, CertExpiringSoon sev2, TlsProbeFailing sev2, CrowdSecBans sev3, SeawallStrayListener sev2, SeawallAgentChanges sev2, SeawallPublicSecretPath sev1, SeawallPublicPath sev2, OriginReachable sev2, CaaMissing sev3, CertUnexpectedIssuer sev1, CtWatchStale sev3, CertWatchFailing sev3, PublishedPortUnguarded sev2, TemplateAccountActive sev2, SshdLoginOpened sev1, SshdLoginOpen sev1, Uid0Account sev1, SystemSecretReadable sev2, PackageFileChanged sev1, PackageCheckBlind sev2, CloudflareOriginPullsPending sev3, CloudflareOriginPullsCertExpiring sev3, AuditPipelineStale sev1 |
updates | Image updates and the CVEs they fix | SecurityUpdateCritical sev1, SecurityUpdateHigh sev2, RebuildNeeded sev2, SeawallImageCritical sev1, SeawallImageHigh sev3, UpdatesStale sev2, NewMajorAvailable sev3, UpdatesPending sev3, UpdatesErrors sev3, AutoUpdateRolledBack sev1, AutoUpdateRollbackFailed sev1, AutoUpdateNeedsPerson sev3, AutoUpdatePolicyWidened sev3, AutoUpdateNight sev3 |
backups | Database and data backups and the config capture | DbBackupStale sev2, DbBackupFailing sev2, DbBackupNotRunning sev2, DataBackupStale sev2, DataBackupFailing sev2, DataBackupRestoreTestFailing sev2, DataBackupDestinationUnreachable sev2, DataBackupOlderImage sev3, ConfigCaptureStale sev2, ConfigCaptureRefused sev2, ConfigNotPushed sev2, ConfigUncovered sev3, ConfigDrift sev3, DriftCheckFailing sev3 |
logs | Critical files, accounts, SSH logins and sudo, read from the logs | SshLoginFromUnknownIp sev0, CriticalFileChanged sev0, AccountChanged sev0, AccountChangedUnchecked sev0, CriticalFileChangedDuringUpgrade sev3, ScheduledTaskChanged sev1, ScheduledTaskChangedUnchecked sev1, SeawallChange sev3, SeawallChangeUnattributed sev1, AdminLoginBruteForce sev1, SiteErrors sev1, LogPipelineStale sev1, SshLogins sev3, SudoUsage sev3, Fail2banBans sev3 |
The security pack's sev2 alerts OriginReachable, PublishedPortUnguarded and TemplateAccountActive each follow one row of the security list. Origins, accounts and patching says what each row checks and when it can't tell.
Every alert
Each alert has a page with what it reads, what decides it and its first command. Decided from names what an alert's answer rests on. The security page's trust model says which of those your account could change.
| Alert | Severity | Pack | Says |
|---|---|---|---|
RootDiskCritical | sev0 | host | Root disk N% used |
DiskHigh | sev1 | host | Disk <mountpoint> N% used |
MemoryLow | sev1 | host | Only N% of memory is free |
ScrapeTargetDown | sev2 | host | Prometheus can't reach <job> |
RaidDegraded | sev0 | host | RAID <device> has a failed or missing disk |
DiskFillingFast | sev1 | host | Disk <mountpoint> is under 25% free and on course to fill within a day |
TmpInodesLow | sev1 | host | /tmp inodes N% used |
RootInodesLow | sev2 | host | Root disk inodes N% used |
SystemdUnitFailed | sev2 | host | The systemd unit <unit> failed |
SecurityUpdatesPending | sev2 | host | N security updates have waited over a day |
NvmeHot | sev2 | host | <chip> is at N°C |
ClockNotSynced | sev2 | host | The clock isn't synchronised |
SeawallFactsStale | sev1 | host | seawall facts last ran Ns ago |
SeawallFactsIncomplete | sev2 | host | seawall facts could not read <collector> or the seawall facts run fails |
SeawallRebootPending | sev3 | host | A reboot has been required for N |
HostUpgradeWorse | sev1 | host | After <package> <from> to <to>: NN |
DockerDownAfterUpgrade | sev1 | host | Docker down 3 minutes after <package> <from> to <to>: <result> |
DriveCriticalWarning | sev1 | host | <device> (<model>) reports a critical warning or failing health |
DriveMediaErrors | sev1 | host | <device> (<model>) counted N new media errors in the last hour |
DriveSpareLow | sev1 | host | <device> (<model>) has less spare than its threshold |
DriveWearHigh | sev2 | host | <device> (<model>) is N% used |
DriveWearFast | sev3 | host | <device> (<model>) reaches 100% used in N days at the last week's write rate |
DriveUnread | sev2 | host | <device> or the drives can't be read: <why> |
SecurityUpdateCritical | sev1 | updates | Image update for <container> (<candidate>) fixes N Critical CVEs |
SecurityUpdateHigh | sev2 | updates | Image update for <container> (<candidate>) fixes N High CVEs |
RebuildNeeded | sev2 | updates | Image rebuild of <container> on its current base fixes N Critical CVEs |
SeawallImageCritical | sev1 | updates | seawall's <container> image has N Critical CVEs with a fix upstream: take the next seawall release that moves its pins |
SeawallImageHigh | sev3 | updates | seawall's <container> image has N High CVEs with a fix upstream |
UpdatesStale | sev2 | updates | The update check has gone N hours without completing a run |
NewMajorAvailable | sev3 | updates | N new image majors: <detail> |
UpdatesPending | sev3 | updates | N image updates, no Critical or High fixes: <detail> |
UpdatesErrors | sev3 | updates | The update check could not check N containers: <detail> |
AutoUpdateRolledBack | sev1 | updates | <container> rolled back after the night's update of <stack> |
AutoUpdateRollbackFailed | sev1 | updates | The night's rollback of <stack> failed: <container> runs the update |
AutoUpdateNeedsPerson | sev3 | updates | The data gate refused the night's update of <stack>, which moves <container> |
AutoUpdatePolicyWidened | sev3 | updates | Automatic updates widened: <detail> |
AutoUpdateNight | sev3 | updates | Last night's automatic updates: N <result> |
ContainerDown | sev1 | containers | Container <name> (<project>) is not running |
ContainerUnhealthy | sev1 | containers | Container <name> (<project>) is unhealthy |
ContainerRestartLoop | sev1 | containers | Container <name> restarted N times in 15 minutes |
SeawallContainerFactsFailing | sev1 | containers | seawall facts can't read Docker, so a container that stops wouldn't alert |
ContainerNearMemoryLimit | sev3 | containers | Container <name> has used over 90% of its memory limit for 15 minutes (N) |
DockerSocketStale | sev2 | containers | Container <name> holds the Docker socket from before the engine's restart and can't reach Docker |
SeawallChangeFinding | sev2 | containers | <project>: refused since its last deploy (N) |
SeawallDeployWorse | sev2 | containers | <stack> is worse since its last change: <what> |
SeawallHostMount | sev2 | containers | Host mounts no compose file accepts: N |
SeawallSocketMount | sev2 | containers | Containers that reach the Docker socket without a proxy: N |
SeawallPrivilegedContainer | sev2 | containers | Privileged containers nothing accepts: N |
SeawallSecretsReadable | sev2 | containers | Secret-looking files other accounts can read: N |
SeawallWorldWritable | sev3 | containers | Files or folders anyone can write in the stacks folder: N |
SeawallRunUserWidened | sev1 | containers | A stack's run user can be logged in as or widened: N ways |
SeawallStackDataSetid | sev2 | containers | Setuid, setgid or capability files in stack folders: N |
SeawallOrphanStackData | sev3 | containers | Stack data no stack owns |
FirewallWeakened | sev0 | security | Firewall weakened: the DOCKER-USER rule <rule> is gone or ufw is off |
FirewallFactsFailing | sev1 | security | The firewall facts are failing, so FirewallWeakened is blind |
UnexpectedPublishedPort | sev1 | security | Container <container> publishes <port>/<proto> to the network, and the allowlist doesn't expect it |
UnexpectedPrivilegedContainer | sev1 | security | Container <container> runs privileged, and the allowlist doesn't expect it |
CrowdSecDown | sev1 | security | CrowdSec is down: nothing is detected and nothing new is banned |
CrowdSecNotParsing | sev1 | security | CrowdSec has read no log lines for 30 minutes, so it detects nothing |
CrowdsecHubFailing | sev3 | security | The last CrowdSec hub upgrade failed: CrowdSec keeps running the set it had |
CrowdsecHubStale | sev3 | security | No CrowdSec hub upgrade has worked in N days |
CrowdSecBouncerStale | sev1 | security | The firewall bouncer hasn't asked CrowdSec for bans in 5 minutes, so none are enforced |
CertExpiringSoon | sev2 | security | The certificate for <domain> expires in N days |
TlsProbeFailing | sev2 | security | The proxy serves no certificate seawall can read for <domain> |
CrowdSecBans | sev3 | security | CrowdSec banned up to N addresses in the last day |
SeawallStrayListener | sev2 | security | Ports left listening by a process started by hand: N |
SeawallAgentChanges | sev2 | security | Changes an agent made through seawall in the last day: N |
SeawallPublicSecretPath | sev1 | security | https://<domain><path> answers the internet with what it leaks |
SeawallPublicPath | sev2 | security | https://<domain><path> answers the internet with what should be private |
OriginReachable | sev2 | security | <domain> is behind Cloudflare, but anyone can reach it at this host directly. Keep 80 and 443 to Cloudflare's ranges in DOCKER-USER, or turn on Authenticated Origin Pulls. |
CaaMissing | sev3 | security | <zone> has no CAA that limits which CA may issue for it, so any CA can. Add the records `seawall security caa` prints. |
CertUnexpectedIssuer | sev1 | security | <issuer> issued a certificate for <name>, and <zone>'s CAA names another CA. If nobody here asked for it, revoke it through that CA and find out who could. |
CtWatchStale | sev3 | security | The certificates logged for <zone> haven't been read for over two days, so a new one would go unseen. |
CertWatchFailing | sev3 | security | The <collector> facts are failing, so the certificate checks are reading old news. |
PublishedPortUnguarded | sev2 | security | <container> publishes <port> to anyone, and nothing in DOCKER-USER stops it. Publish it on 127.0.0.1, or drop it in DOCKER-USER. |
TemplateAccountActive | sev2 | security | <user>, a cloud image template account, can still log in and is privileged. Remove its keys and end it with seawall host fix expire <user>, or delete it (a locked password alone leaves keys working). |
SshdLoginOpened | sev1 | security | sshd now has <setting> on, which it didn't an hour ago: someone can log in with less than a key. Unless it was you, set it to no in a drop-in under /etc/ssh/sshd_config.d/ and reload ssh. This page ends after the hour; SshdLoginOpen stays while the setting is on. |
SshdLoginOpen | sev1 | security | sshd has <setting> on: someone can log in with less than a key. Set it to no in a drop-in under /etc/ssh/sshd_config.d/ and reload ssh, or `seawall host fix sshd`. |
Uid0Account | sev1 | security | <user> has UID 0, so it is root. Unless someone made it on purpose, it is a backdoor: look at what it ran, then remove it with userdel. |
SystemSecretReadable | sev2 | security | <path> is readable by more than CIS allows: password hashes or a host key anyone could copy. Set it back with chmod as root (shadow 0640 root:shadow, a host key 0600). |
PackageFileChanged | sev1 | security | <path> of <package> is not the file dpkg put there. Unless you changed it, something replaced it: look at what did (the audit log), then sudo apt install --reinstall <package>. |
PackageCheckBlind | sev2 | security | No package checked against dpkg for 2 hours: a replaced sshd, sudo or Docker binary pages nothing. |
CloudflareOriginPullsPending | sev3 | security | Authenticated Origin Pulls for <host> wait on steps in Cloudflare's dashboard |
CloudflareOriginPullsCertExpiring | sev3 | security | The certificate Cloudflare presents for <host> ends in N; past it, Cloudflare can't reach the site |
DbBackupStale | sev2 | backups | No restore-tested dump of <container> for 36 hours |
DbBackupFailing | sev2 | backups | The last scheduled database backup did not fully succeed (a dump or its restore test failed) |
DbBackupNotRunning | sev2 | backups | The scheduled database backup has not run for 30 hours (timer stopped, or runs hang or are killed) |
DataBackupStale | sev2 | backups | No data backup has worked for every stack in N hours |
DataBackupFailing | sev2 | backups | The last data backup failed for a stack, its retention or its check |
DataBackupRestoreTestFailing | sev2 | backups | The weekly restore test failed, or hasn't passed in 8 days |
DataBackupDestinationUnreachable | sev2 | backups | The data backup repository didn't answer, so nothing was backed up |
DataBackupOlderImage | sev3 | backups | The data backup ran on an older restic image, as this version's couldn't be built |
ConfigCaptureStale | sev2 | backups | No config capture has worked in N hours |
ConfigCaptureRefused | sev2 | backups | The config capture was refused: something in it looked like a secret, and nothing was committed |
ConfigNotPushed | sev2 | backups | The config capture committed but couldn't push: the copy off this host is behind |
ConfigUncovered | sev3 | backups | N config paths the last capture couldn't hold |
ConfigDrift | sev3 | backups | N things on this host differ from the config repo's last commit |
DriftCheckFailing | sev3 | backups | The config drift check hasn't run in N hours |
SiteDown | sev1 | host | <domain> doesn't answer as a visitor reaches it |
SiteSlow | sev3 | host | <domain> takes Ns to answer |
SshLoginFromUnknownIp | sev0 | logs | An SSH login from <ip>, which [logging] ssh_from doesn't list |
CriticalFileChanged | sev0 | logs | A security-critical file changed (audit key <key>), by <comm> as <user> or with no login user, N writes |
AccountChanged | sev0 | logs | <comm> changed the accounts as <user> or with no login user, and seawall recorded no account change |
AccountChangedUnchecked | sev0 | logs | <comm> changed the accounts as <user> or with no login user, and seawall recorded no account change (root's count is stale) |
CriticalFileChangedDuringUpgrade | sev3 | logs | A security-critical file changed during a package upgrade (audit key <key>), by <comm> as <user> or with no login user |
ScheduledTaskChanged | sev1 | logs | Cron or systemd unit files changed outside a package update, by <comm> as <user> or with no login user |
ScheduledTaskChangedUnchecked | sev1 | logs | Cron or systemd unit files changed outside a package update, by <comm> as <user> or with no login user (root's count is stale) |
SeawallChange | sev3 | logs | <comm> wrote files under audit key <key> while seawall recorded a change |
SeawallChangeUnattributed | sev1 | logs | seawall changed something N times from outside a terminal, a login session, the owner's own units or its timers, in the last 15 or 25 minutes |
AdminLoginBruteForce | sev1 | logs | N failed admin logins on <vhost> in 10 minutes |
SiteErrors | sev1 | logs | <vhost> answers over 5% of requests with a server error |
LogPipelineStale | sev1 | logs | No journal lines reached Loki in 15 minutes, so the log alerts are blind |
SshLogins | sev3 | logs | N SSH logins from <ip> in the last day |
SudoUsage | sev3 | logs | <user> ran N sudo commands in the last day; seawall init says N were its own |
Fail2banBans | sev3 | logs | fail2ban's <jail> jail banned N times in the last day |
AuditPipelineStale | sev1 | security | No audit records reached Loki in 15 minutes, so the audit alerts are blind |
Probes
seawall proxy probes
seawall proxy probes add blog.example.org --yes
seawall proxy probes rm blog.example.org --yes
A probe checks a public host as a visitor reaches it, through public DNS, the proxy and TLS, every 60 seconds. Probes run on the blackbox exporter in seawall's monitoring stack, read-only with no capabilities and no published port. A monitoring stack init wrote before the exporter existed gets it on the first proxy probes add. SiteDown (sev1) fires after 3 minutes down and SiteSlow (sev3) after 15 minutes of slow answers. Only a domain the proxy routes can be probed.