Alerts and silences

Read what an alert found, run its fix and silence planned work.

Alerts

seawall alerts                          # what fires
seawall alerts ContainerDown            # one alert with what it found and what fixes it
seawall silence add nextcloud --for 2h --comment "upgrade" --yes
seawall silence alert DiskHigh --label mountpoint=/ --for 1h --yes
seawall silence ls
seawall silence rm <id> --yes
seawall reboot-prep --yes               # 15 minutes for a planned reboot

alerts <name> reads the alert's cause live. Found lists what it saw. Do lists the commands that fix it, each a dry run first unless it acts at once. In the command centre each Do line is a number key, marked now where it acts at once. Under CloudflareOriginPullsCertExpiring, proxy aop renew and, for a host set up by hand, proxy aop HOST --guided act at once and have no mark.

init writes a rule file for each pack. The host, containers, security, updates and backups packs come with the monitoring stack. The logs pack reads Loki and comes with the logging step. Its sev0 SSH login alert needs [logging] ssh_from. The audit step adds rules-seawall-audit.yaml, which holds AuditPipelineStale. AdminLoginBruteForce and SiteErrors read the proxy's logs, so they exist only when Nginx Proxy Manager is the proxy.

Daily findings (SecurityUpdateCritical, SeawallImageCritical) carry the label page=daily. Where seawall routes the alerts, it pages them once a day within notifications.page_hours (08:00-23:59 by default). A finding that fires inside the window pages at once, and one that fires outside it pages when the window opens. Where you route alerts yourself, give page=daily a route placed before your severity=sev1 route, with active_time_intervals set to [seawall-page-hours] and repeat_interval set to 23h55m. The first matching route wins, and both daily rules are sev1.

A silence never covers a sev0 alert. silence add refuses a name that is neither a container nor a stack. silence alert takes any alert name and labels, so check the name with seawall alerts <name> first. A silence lasts from 1 minute to 30 days.

The Alerts tab with the Firing table and the Silenced box
The Alerts tab. Silences sit below the firing alerts on one cursor.

Where alerts go

The Pages and Rollup rows in seawall config set where alerts go: Discord, Slack, Telegram, Pushover, ntfy, Gotify, Matrix, a webhook or email. Pages takes sev0 and sev1 at once and daily findings within notifications.page_hours. Grafana's DatasourceError and DatasourceNoData, and any alert with no sev label, go to Pages too. Rollup takes sev2 and sev3 in the 09:00-09:15 window on the host's clock. With no Rollup row set, they go to Pages. Each secret is typed hidden and kept where only Grafana reads it. A row can send a test or be removed. seawall notify test sends one to Pages, Rollup or both through Grafana, which holds the secrets. seawall never reads or prints them.

Email goes through the host's one mail server, set in Grafana's own settings with its password in a file. Grafana must use STARTTLS, so a server that doesn't offer it fails and the password never crosses in clear. notifications.smtp.plain allows clear text when the server offers no TLS. Port 465 is TLS from the start.

Rule packs

monitoring.packs picks the rule packs seawall's monitoring keeps, from host, containers, security, updates, backups and logs. The old names health, audit and probes read as the packs they became. A pack turned off keeps its sev0 rules. One that holds sev1 rules names them and goes off only with the pack's name typed. A pack rule whose title the host already has in a rule file of its own is left out, so nothing pages twice. init keeps each file to this version's. One you edited is left as it is when seawall has a record of its last copy, and replaced, with yours kept in ~/backups/seawall/rules/, when it hasn't.

The packs hold 116 rules.

PackAlerts onRules
hostDisk, memory, RAID, drive wear and health, NVMe heat, failed units, the clock, seawall's facts, reboots, sites downRootDiskCritical sev0, DiskHigh sev1, MemoryLow sev1, ScrapeTargetDown sev2, RaidDegraded sev0, DiskFillingFast sev1, TmpInodesLow sev1, RootInodesLow sev2, SystemdUnitFailed sev2, SecurityUpdatesPending sev2, NvmeHot sev2, ClockNotSynced sev2, SeawallFactsStale sev1, SeawallFactsIncomplete sev2, SeawallRebootPending sev3, HostUpgradeWorse sev1, DockerDownAfterUpgrade sev1, DriveCriticalWarning sev1, DriveMediaErrors sev1, DriveSpareLow sev1, DriveWearHigh sev2, DriveWearFast sev3, DriveUnread sev2, SiteDown sev1, SiteSlow sev3
containersContainers down, unhealthy or restarting, and risky mounts and privilegesContainerDown sev1, ContainerUnhealthy sev1, ContainerRestartLoop sev1, SeawallContainerFactsFailing sev1, ContainerNearMemoryLimit sev3, DockerSocketStale sev2, SeawallChangeFinding sev2, SeawallDeployWorse sev2, SeawallHostMount sev2, SeawallSocketMount sev2, SeawallPrivilegedContainer sev2, SeawallSecretsReadable sev2, SeawallWorldWritable sev3, SeawallRunUserWidened sev1, SeawallStackDataSetid sev2, SeawallOrphanStackData sev3
securityThe firewall, CrowdSec, certificates, exposed ports and paths, the audit pipelineFirewallWeakened sev0, FirewallFactsFailing sev1, UnexpectedPublishedPort sev1, UnexpectedPrivilegedContainer sev1, CrowdSecDown sev1, CrowdSecNotParsing sev1, CrowdsecHubFailing sev3, CrowdsecHubStale sev3, CrowdSecBouncerStale sev1, CertExpiringSoon sev2, TlsProbeFailing sev2, CrowdSecBans sev3, SeawallStrayListener sev2, SeawallAgentChanges sev2, SeawallPublicSecretPath sev1, SeawallPublicPath sev2, OriginReachable sev2, CaaMissing sev3, CertUnexpectedIssuer sev1, CtWatchStale sev3, CertWatchFailing sev3, PublishedPortUnguarded sev2, TemplateAccountActive sev2, SshdLoginOpened sev1, SshdLoginOpen sev1, Uid0Account sev1, SystemSecretReadable sev2, PackageFileChanged sev1, PackageCheckBlind sev2, CloudflareOriginPullsPending sev3, CloudflareOriginPullsCertExpiring sev3, AuditPipelineStale sev1
updatesImage updates and the CVEs they fixSecurityUpdateCritical sev1, SecurityUpdateHigh sev2, RebuildNeeded sev2, SeawallImageCritical sev1, SeawallImageHigh sev3, UpdatesStale sev2, NewMajorAvailable sev3, UpdatesPending sev3, UpdatesErrors sev3, AutoUpdateRolledBack sev1, AutoUpdateRollbackFailed sev1, AutoUpdateNeedsPerson sev3, AutoUpdatePolicyWidened sev3, AutoUpdateNight sev3
backupsDatabase and data backups and the config captureDbBackupStale sev2, DbBackupFailing sev2, DbBackupNotRunning sev2, DataBackupStale sev2, DataBackupFailing sev2, DataBackupRestoreTestFailing sev2, DataBackupDestinationUnreachable sev2, DataBackupOlderImage sev3, ConfigCaptureStale sev2, ConfigCaptureRefused sev2, ConfigNotPushed sev2, ConfigUncovered sev3, ConfigDrift sev3, DriftCheckFailing sev3
logsCritical files, accounts, SSH logins and sudo, read from the logsSshLoginFromUnknownIp sev0, CriticalFileChanged sev0, AccountChanged sev0, AccountChangedUnchecked sev0, CriticalFileChangedDuringUpgrade sev3, ScheduledTaskChanged sev1, ScheduledTaskChangedUnchecked sev1, SeawallChange sev3, SeawallChangeUnattributed sev1, AdminLoginBruteForce sev1, SiteErrors sev1, LogPipelineStale sev1, SshLogins sev3, SudoUsage sev3, Fail2banBans sev3

The security pack's sev2 alerts OriginReachable, PublishedPortUnguarded and TemplateAccountActive each follow one row of the security list. Origins, accounts and patching says what each row checks and when it can't tell.

Every alert

Each alert has a page with what it reads, what decides it and its first command. Decided from names what an alert's answer rests on. The security page's trust model says which of those your account could change.

AlertSeverityPackSays
RootDiskCriticalsev0hostRoot disk N% used
DiskHighsev1hostDisk <mountpoint> N% used
MemoryLowsev1hostOnly N% of memory is free
ScrapeTargetDownsev2hostPrometheus can't reach <job>
RaidDegradedsev0hostRAID <device> has a failed or missing disk
DiskFillingFastsev1hostDisk <mountpoint> is under 25% free and on course to fill within a day
TmpInodesLowsev1host/tmp inodes N% used
RootInodesLowsev2hostRoot disk inodes N% used
SystemdUnitFailedsev2hostThe systemd unit <unit> failed
SecurityUpdatesPendingsev2hostN security updates have waited over a day
NvmeHotsev2host<chip> is at N°C
ClockNotSyncedsev2hostThe clock isn't synchronised
SeawallFactsStalesev1hostseawall facts last ran Ns ago
SeawallFactsIncompletesev2hostseawall facts could not read <collector> or the seawall facts run fails
SeawallRebootPendingsev3hostA reboot has been required for N
HostUpgradeWorsesev1hostAfter <package> <from> to <to>: NN
DockerDownAfterUpgradesev1hostDocker down 3 minutes after <package> <from> to <to>: <result>
DriveCriticalWarningsev1host<device> (<model>) reports a critical warning or failing health
DriveMediaErrorssev1host<device> (<model>) counted N new media errors in the last hour
DriveSpareLowsev1host<device> (<model>) has less spare than its threshold
DriveWearHighsev2host<device> (<model>) is N% used
DriveWearFastsev3host<device> (<model>) reaches 100% used in N days at the last week's write rate
DriveUnreadsev2host<device> or the drives can't be read: <why>
SecurityUpdateCriticalsev1updatesImage update for <container> (<candidate>) fixes N Critical CVEs
SecurityUpdateHighsev2updatesImage update for <container> (<candidate>) fixes N High CVEs
RebuildNeededsev2updatesImage rebuild of <container> on its current base fixes N Critical CVEs
SeawallImageCriticalsev1updatesseawall's <container> image has N Critical CVEs with a fix upstream: take the next seawall release that moves its pins
SeawallImageHighsev3updatesseawall's <container> image has N High CVEs with a fix upstream
UpdatesStalesev2updatesThe update check has gone N hours without completing a run
NewMajorAvailablesev3updatesN new image majors: <detail>
UpdatesPendingsev3updatesN image updates, no Critical or High fixes: <detail>
UpdatesErrorssev3updatesThe update check could not check N containers: <detail>
AutoUpdateRolledBacksev1updates<container> rolled back after the night's update of <stack>
AutoUpdateRollbackFailedsev1updatesThe night's rollback of <stack> failed: <container> runs the update
AutoUpdateNeedsPersonsev3updatesThe data gate refused the night's update of <stack>, which moves <container>
AutoUpdatePolicyWidenedsev3updatesAutomatic updates widened: <detail>
AutoUpdateNightsev3updatesLast night's automatic updates: N <result>
ContainerDownsev1containersContainer <name> (<project>) is not running
ContainerUnhealthysev1containersContainer <name> (<project>) is unhealthy
ContainerRestartLoopsev1containersContainer <name> restarted N times in 15 minutes
SeawallContainerFactsFailingsev1containersseawall facts can't read Docker, so a container that stops wouldn't alert
ContainerNearMemoryLimitsev3containersContainer <name> has used over 90% of its memory limit for 15 minutes (N)
DockerSocketStalesev2containersContainer <name> holds the Docker socket from before the engine's restart and can't reach Docker
SeawallChangeFindingsev2containers<project>: refused since its last deploy (N)
SeawallDeployWorsesev2containers<stack> is worse since its last change: <what>
SeawallHostMountsev2containersHost mounts no compose file accepts: N
SeawallSocketMountsev2containersContainers that reach the Docker socket without a proxy: N
SeawallPrivilegedContainersev2containersPrivileged containers nothing accepts: N
SeawallSecretsReadablesev2containersSecret-looking files other accounts can read: N
SeawallWorldWritablesev3containersFiles or folders anyone can write in the stacks folder: N
SeawallRunUserWidenedsev1containersA stack's run user can be logged in as or widened: N ways
SeawallStackDataSetidsev2containersSetuid, setgid or capability files in stack folders: N
SeawallOrphanStackDatasev3containersStack data no stack owns
FirewallWeakenedsev0securityFirewall weakened: the DOCKER-USER rule <rule> is gone or ufw is off
FirewallFactsFailingsev1securityThe firewall facts are failing, so FirewallWeakened is blind
UnexpectedPublishedPortsev1securityContainer <container> publishes <port>/<proto> to the network, and the allowlist doesn't expect it
UnexpectedPrivilegedContainersev1securityContainer <container> runs privileged, and the allowlist doesn't expect it
CrowdSecDownsev1securityCrowdSec is down: nothing is detected and nothing new is banned
CrowdSecNotParsingsev1securityCrowdSec has read no log lines for 30 minutes, so it detects nothing
CrowdsecHubFailingsev3securityThe last CrowdSec hub upgrade failed: CrowdSec keeps running the set it had
CrowdsecHubStalesev3securityNo CrowdSec hub upgrade has worked in N days
CrowdSecBouncerStalesev1securityThe firewall bouncer hasn't asked CrowdSec for bans in 5 minutes, so none are enforced
CertExpiringSoonsev2securityThe certificate for <domain> expires in N days
TlsProbeFailingsev2securityThe proxy serves no certificate seawall can read for <domain>
CrowdSecBanssev3securityCrowdSec banned up to N addresses in the last day
SeawallStrayListenersev2securityPorts left listening by a process started by hand: N
SeawallAgentChangessev2securityChanges an agent made through seawall in the last day: N
SeawallPublicSecretPathsev1securityhttps://<domain><path> answers the internet with what it leaks
SeawallPublicPathsev2securityhttps://<domain><path> answers the internet with what should be private
OriginReachablesev2security<domain> is behind Cloudflare, but anyone can reach it at this host directly. Keep 80 and 443 to Cloudflare's ranges in DOCKER-USER, or turn on Authenticated Origin Pulls.
CaaMissingsev3security<zone> has no CAA that limits which CA may issue for it, so any CA can. Add the records `seawall security caa` prints.
CertUnexpectedIssuersev1security<issuer> issued a certificate for <name>, and <zone>'s CAA names another CA. If nobody here asked for it, revoke it through that CA and find out who could.
CtWatchStalesev3securityThe certificates logged for <zone> haven't been read for over two days, so a new one would go unseen.
CertWatchFailingsev3securityThe <collector> facts are failing, so the certificate checks are reading old news.
PublishedPortUnguardedsev2security<container> publishes <port> to anyone, and nothing in DOCKER-USER stops it. Publish it on 127.0.0.1, or drop it in DOCKER-USER.
TemplateAccountActivesev2security<user>, a cloud image template account, can still log in and is privileged. Remove its keys and end it with seawall host fix expire <user>, or delete it (a locked password alone leaves keys working).
SshdLoginOpenedsev1securitysshd now has <setting> on, which it didn't an hour ago: someone can log in with less than a key. Unless it was you, set it to no in a drop-in under /etc/ssh/sshd_config.d/ and reload ssh. This page ends after the hour; SshdLoginOpen stays while the setting is on.
SshdLoginOpensev1securitysshd has <setting> on: someone can log in with less than a key. Set it to no in a drop-in under /etc/ssh/sshd_config.d/ and reload ssh, or `seawall host fix sshd`.
Uid0Accountsev1security<user> has UID 0, so it is root. Unless someone made it on purpose, it is a backdoor: look at what it ran, then remove it with userdel.
SystemSecretReadablesev2security<path> is readable by more than CIS allows: password hashes or a host key anyone could copy. Set it back with chmod as root (shadow 0640 root:shadow, a host key 0600).
PackageFileChangedsev1security<path> of <package> is not the file dpkg put there. Unless you changed it, something replaced it: look at what did (the audit log), then sudo apt install --reinstall <package>.
PackageCheckBlindsev2securityNo package checked against dpkg for 2 hours: a replaced sshd, sudo or Docker binary pages nothing.
CloudflareOriginPullsPendingsev3securityAuthenticated Origin Pulls for <host> wait on steps in Cloudflare's dashboard
CloudflareOriginPullsCertExpiringsev3securityThe certificate Cloudflare presents for <host> ends in N; past it, Cloudflare can't reach the site
DbBackupStalesev2backupsNo restore-tested dump of <container> for 36 hours
DbBackupFailingsev2backupsThe last scheduled database backup did not fully succeed (a dump or its restore test failed)
DbBackupNotRunningsev2backupsThe scheduled database backup has not run for 30 hours (timer stopped, or runs hang or are killed)
DataBackupStalesev2backupsNo data backup has worked for every stack in N hours
DataBackupFailingsev2backupsThe last data backup failed for a stack, its retention or its check
DataBackupRestoreTestFailingsev2backupsThe weekly restore test failed, or hasn't passed in 8 days
DataBackupDestinationUnreachablesev2backupsThe data backup repository didn't answer, so nothing was backed up
DataBackupOlderImagesev3backupsThe data backup ran on an older restic image, as this version's couldn't be built
ConfigCaptureStalesev2backupsNo config capture has worked in N hours
ConfigCaptureRefusedsev2backupsThe config capture was refused: something in it looked like a secret, and nothing was committed
ConfigNotPushedsev2backupsThe config capture committed but couldn't push: the copy off this host is behind
ConfigUncoveredsev3backupsN config paths the last capture couldn't hold
ConfigDriftsev3backupsN things on this host differ from the config repo's last commit
DriftCheckFailingsev3backupsThe config drift check hasn't run in N hours
SiteDownsev1host<domain> doesn't answer as a visitor reaches it
SiteSlowsev3host<domain> takes Ns to answer
SshLoginFromUnknownIpsev0logsAn SSH login from <ip>, which [logging] ssh_from doesn't list
CriticalFileChangedsev0logsA security-critical file changed (audit key <key>), by <comm> as <user> or with no login user, N writes
AccountChangedsev0logs<comm> changed the accounts as <user> or with no login user, and seawall recorded no account change
AccountChangedUncheckedsev0logs<comm> changed the accounts as <user> or with no login user, and seawall recorded no account change (root's count is stale)
CriticalFileChangedDuringUpgradesev3logsA security-critical file changed during a package upgrade (audit key <key>), by <comm> as <user> or with no login user
ScheduledTaskChangedsev1logsCron or systemd unit files changed outside a package update, by <comm> as <user> or with no login user
ScheduledTaskChangedUncheckedsev1logsCron or systemd unit files changed outside a package update, by <comm> as <user> or with no login user (root's count is stale)
SeawallChangesev3logs<comm> wrote files under audit key <key> while seawall recorded a change
SeawallChangeUnattributedsev1logsseawall changed something N times from outside a terminal, a login session, the owner's own units or its timers, in the last 15 or 25 minutes
AdminLoginBruteForcesev1logsN failed admin logins on <vhost> in 10 minutes
SiteErrorssev1logs<vhost> answers over 5% of requests with a server error
LogPipelineStalesev1logsNo journal lines reached Loki in 15 minutes, so the log alerts are blind
SshLoginssev3logsN SSH logins from <ip> in the last day
SudoUsagesev3logs<user> ran N sudo commands in the last day; seawall init says N were its own
Fail2banBanssev3logsfail2ban's <jail> jail banned N times in the last day
AuditPipelineStalesev1securityNo audit records reached Loki in 15 minutes, so the audit alerts are blind

Probes

seawall proxy probes
seawall proxy probes add blog.example.org --yes
seawall proxy probes rm blog.example.org --yes

A probe checks a public host as a visitor reaches it, through public DNS, the proxy and TLS, every 60 seconds. Probes run on the blackbox exporter in seawall's monitoring stack, read-only with no capabilities and no published port. A monitoring stack init wrote before the exporter existed gets it on the first proxy probes add. SiteDown (sev1) fires after 3 minutes down and SiteSlow (sev3) after 15 minutes of slow answers. Only a domain the proxy routes can be probed.