Multi-Agent Alert Architecture: Deduplicating Alerts When Every Agent Can Report

My gateway and the Telegram bridge have died together more than once, both with SIGTERM, both inside the same minute. launchd brought both back before anyone was awake. What was left for a human was two exits with status 143 and two fresh PIDs, and later a morning digest describing the night in its own words. One cause, on one Mac mini, and a pile of separate text about it, none of which shared a single identifier with any other.

The process count did it, every time.

What the Standard Deduplication Docs Agree On

Read the alert deduplication pages from incident.io, Rootly and Cortex XDR and you get a consistent model. Every alert carries a deduplication key. If a second alert arrives with the same key while the first is still open, no new alert is created, and the first one gets a counter bumped instead (Rootly shows it as a small ×3 badge). Resolving the alert clears the key, so the next occurrence opens fresh. Cortex adds a sliding one-hour window. Rootly warns against building keys from free-form text or timestamps and tells you to use something stable, like a monitor ID or an alarm ARN.

That last warning is the one an agent fleet violates by default. Agents produce free-form text. It is the only thing they produce. An agent that noticed the bridge was down will say so differently at 3:02 and 3:04, and two different agents will never phrase it the same way at all, so a key hashed from the message body dedupes nothing.

Take the Key From the Check, Never From the Sentence

The alert architecture on this site already forbids agents from writing warnings freehand. A warning line can only be rendered if it names a check that actually ran recently. That rule was built to stop invented alerts, and it turns out to be the thing that makes deduplication possible across agents, because every alert now arrives carrying a check_id that some program, rather than some model, chose.

# alert_key.py: identical for every agent that reports the same fact
def alert_key(alert):
    return f"{alert['check_id']}:{alert['subject']}:{alert['state']}"

# 'gateway-health:com.openclaw.gateway:down'
# 'uid-procs:agent-account:over_gate'

Three parts, all of them machine-chosen. The agent's prose rides along as an attachment on the alert and is shown to the human, but it plays no part in identity. Two agents that both notice the gateway is down produce one key. The watchdog that recovered it produces the same key too, with a different state.

Storage is one table with a unique key and a separate list of who reported it. This is the same trick the work queue uses for jobs, where two producers asking for digest.daily:2026-09-30 collapse into one row.

CREATE TABLE alerts (
  key         TEXT PRIMARY KEY,
  cause_id    TEXT,              -- host-level incident, if one is open
  first_seen  INTEGER NOT NULL,
  last_seen   INTEGER NOT NULL,
  count       INTEGER NOT NULL DEFAULT 1,
  state       TEXT NOT NULL      -- open | resolved
);
CREATE TABLE alert_reporters (
  key    TEXT NOT NULL REFERENCES alerts(key),
  agent  TEXT NOT NULL,
  note   TEXT,                   -- the agent's own words, kept for the human
  at     INTEGER NOT NULL
);

INSERT INTO alerts (key, first_seen, last_seen, state)
VALUES (:key, :now, :now, 'open')
ON CONFLICT(key) DO UPDATE
  SET last_seen = :now, count = count + 1;

Group by Cause, Because Services Die Together

Key deduplication handles one service failing loudly. It does nothing for the gateway and the bridge failing in the same minute, because those are two honest, different keys. The incident tools call this grouping and treat it as a separate feature. For a fleet on one machine I think grouping is the more important half, since almost every multi-service failure I have seen traces back to the host.

On my mini the cause has been the same each time: the process count for the account the agents run under climbed toward its ceiling, spawns failed with EAGAIN, and long-running services were killed in the scramble. The self-healing piece describes the wrapper that photographs the process table when SIGTERM arrives. That snapshot is the grouping signal. When an alert lands within two minutes of a snapshot showing the agent account over its gate of 550 processes, it gets the snapshot's cause_id, and the human sees one incident with the affected services listed underneath.

Without that evidence, I would rather not group at all. Grouping by time alone is guessing, and a guess that merges two unrelated failures hides the second one until somebody goes looking, which nobody does at 6 a.m.

A Counter Tells You More Than a Repeat

My signal job once re-fired the same five resolved categories for the seventeenth time since July 29, because a report generator kept rewriting the file and dropping the fields that marked them resolved. Each re-fire arrived as fresh news and cost a fresh round of investigation that ended with "already fixed." Seventeen alerts. One problem, and the problem was the reporting.

Had those arrived as a single row with count = 17 and a first_seen of July 29, the conclusion would have been obvious on day three: something keeps reopening this. A count turns noise into a trend you can read, and the digest's fixed format (counts first, same order every morning) is built to show exactly that kind of number moving.

Resolution Belongs to One Writer

The standard docs say resolving an alert clears its key. With one monitoring tool that is fine. With several agents, any one of which might decide the thing is fixed, it is how a real recurrence gets swallowed: agent A marks it resolved, agent B's report arrives a minute later and reopens it, agent A resolves it again.

So agents report and never resolve. Only the check resolves, by running and coming back healthy, and a human can resolve by hand in the signal_state table with a reopen_if condition attached. The learning loop runs on the same discipline from the other side: the harvester checks whether a skill already owns a category before it proposes anything, and on September 4 that check found five skills on disk and wrote nothing new.

A suppression window still has a job here, a narrower one than in the docs. I keep the channel windows from the renderer (fifteen minutes for a page, twenty-six hours for the digest) and suppress repeat notifications for an open key inside them. The row keeps counting the whole time. Suppressing the message and suppressing the record are different decisions, and only the first one is safe.

Where to Start

If your agents write warnings freehand today, deduplication will not save you, so start with the renderer rule and give every alert a check_id. Then add the unique key and the counter. That alone collapses most of the duplicates, because most of them are the same agent saying the same thing on every run.

Cause grouping comes last, and only once you have a snapshot you trust to name the cause. Mine took a few SIGTERM clusters and a wrapper script to earn.

Internal Links & Further Reading

• OpenClaw Alert Architecture →

  The renderer that refuses warnings without a recent check run, which is where the dedup key comes from.

• OpenClaw Self-Healing Architecture →

  The SIGTERM wrapper whose process snapshots let alerts be grouped by cause.

• OpenClaw Work Queue Architecture →

  Dedupe keys and a unique index for jobs, the same pattern applied to work instead of warnings.

• Monitoring and Observability for Agents →

  The checks and metrics that alerts point back at.

FAQ