OpenClaw Completion Verification Architecture: Grading an Agent's Done Report

On February 8 this site shipped with a delivery report that opened with a green checkmark and the words DEPLOYED AND LIVE. Below it sat a verification checklist of twelve items, every one ticked. Navigation links functional. Mobile responsive. Sister site links work. Blog listing shows all 10 articles. The evidence offered for the whole list was a single line near the top: HTTP 200, all pages rendering correctly. Six of those ten articles were stubs, and a second status file in the same repo said so plainly, with a warning sign next to each one.

A 200 means the server answered.

What the Verification Guides Agree On

The top results for verifying AI agent task completion run from roughly 3,300 words to over 5,000, and on the main points they agree. Separate the builder from the grader. Check the state of the system instead of the agent's account of it, and write acceptance criteria before the agent starts. Keep LLM judges on a short leash, because a model grading a model grades generously. The longest of them adds a receipt bound to a commit hash, so a passing test goes stale the moment the code moves.

I agree with all of it. Every one of those pages is about code graded by tests, though, and a fleet says done about plenty of things no test suite touches: a model upgrade, a skill that should load, a warning footer, how a site lays out on a phone. Those claims reach a human through a report, and the report flattens very different kinds of evidence into one identical checkmark. None of the three pages I read spends a paragraph on that step.

Put Every Claim on a Rung

I sort each claim in a done report onto one of four rungs, and the report has to say which.

• written: the file exists, the edit is saved. Proves very little.
• built: the build exited 0, so it parses and type-checks.
• ran here: a probe exercised the behavior somewhere that is not production, like a worktree's dev server or a fresh session.
• seen live: the running production system showed the behavior, read from the system itself and never from a document about it.

The model upgrade from the fleet registry article lived entirely between the first rung and the last. A status doc said an agent had been upgraded on June 30. The launcher hardcoded --model claude-sonnet-5 in two places and won every time. Reading the argv of the running process is one ps command and sits on rung four. Nobody ran it for months, because the doc already had a checkmark.

The best done report in this repo is the overnight note from May 5, and it does the ladder by instinct. It lists the commits. Then, under a heading for what it deliberately left alone, it admits the mobile test was never run: the breakpoints looked fine in code review, but no real browser was pointed at the page at 375 pixels. February's list said "Mobile responsive" with a tick. Same claim, two rungs apart. Only one of the two reports told you which rung you were standing on, and I trust that one more even though it claims less.

A Checkmark Has No Timestamp

That same May night found the footer's Resources column pointing at three anchors, #cost, #multi-agent and #case-studies, that existed nowhere on the site, leftovers from an earlier homepage layout. The nav had two more. I can't tell you whether those links worked on February 8. Neither can the February report, and that is my complaint about it. Its tick carried no date for the check behind it, so "navigation links functional" kept asserting itself straight through the redesign that broke them.

A receipt should carry the time it ran and the hash of what it ran against. The alert renderer already refuses a warning whose check is older than its channel allows (fifteen minutes for a page, twenty-six hours for the digest). Done reports deserve the same treatment:

# done_report.py: a claim prints at the rung its receipt proves, or not at all
def line(claim, receipts, head):
    r = receipts.get(claim["check_id"])
    if r is None:
        return f"NOT CHECKED  {claim['text']}"
    if r["exit"] != 0:
        return f"FAILED       {claim['text']}"
    if r["against"] != head:            # commit or config hash moved
        return f"STALE        {claim['text']}  (proved {r['ran_at']})"
    return f"{r['rung'].upper():<12} {claim['text']}"

# SEEN_LIVE    launcher argv shows the new --model flag
# RAN_HERE     /blog index lists the new post (local next start)
# NOT CHECKED  layout at 375px

Silence counts as failure here, same as in the config apply script that reverts unless its probe passes. A claim with no receipt still appears in the report, in the same weight as a failure, labelled for what it is. The default in most agent output runs the other way, and the cost of that default showed up this spring as a GA4 warning footer that rode along on message after message with no check behind it at all.

Make the Acceptance Check Able to Fail

The guides are right that acceptance comes first. They say much less about acceptance that cannot fail.

The contract that produced this page came with two acceptance commands, test -d $PWD and npm run build. The first one passes in an empty directory. It checks that I am standing somewhere. The build is a real gate on rung two, and the sweep re-runs it on main after merging, which is why the merge lines in this repo's history end in bb auto-sweep, thread ... PASS. Neither command looks at whether the page exists or obeys the eleven voice rules the contract spends most of its words on. A branch could pass both with no new page in it.

One rule, then. Before the agent starts, run its acceptance check against the untouched worktree and watch it fail. A check that passes before any work is done measures the environment. For a content page the cheap version fetches the route from a local server and greps the HTML for the things the contract forbids, which is about two lines of shell and catches more than the build does. The learning loop piece asks the same of skills: open a fresh session and look for the skill's name in what it loaded.

Who Writes What

The agent that did the work gets to write the claims and something that never saw the work writes the receipts, and if a person only reads one of the two it should be the receipts, with the agent's prose stapled behind them like a cover letter nobody is obliged to open.

Where to Start

If your agents end every task with a paragraph of prose, add one required field before anything else: NOT CHECKED, listing each claim in that paragraph with no command behind it. Give an agent the field and the honest answer at least has somewhere to go. The May 5 note did it without being asked, which is the example I keep pointing at.

Rungs and timestamps come next. The renderer that builds reports from receipts can wait until you have enough receipts to make it worth writing, and by then you will know which of your checkmarks were load-bearing and which were just green.

Internal Links & Further Reading

• OpenClaw Config Change Architecture →

  Probes that ask the running system, and an apply script that treats silence as failure.

• OpenClaw Alert Architecture →

  The renderer rule for warnings, which done reports borrow wholesale.

• OpenClaw Fleet Registry Architecture →

  The status doc that said upgraded while the launcher said otherwise.

• OpenClaw Worktree Architecture for Coding Agents →

  The sweep that re-runs the build on main and refuses to round its leak count.

FAQ