A read-only auditor for AI-agent event logs
AI-Watchbird-Sheckley is a small Python module that reads the event log of a multi-agent AI run and reports signs that its oversight is growing, stalling or destroying work. It has no authority: it never blocks, edits or opens anything.
Why audit the oversight loop itself
The paper reviewed existing guards, step and budget counters, and model-judge evaluators on 14 September 2026. None of the tools in the public documentation reviewed takes the oversight loop itself as the unit of measurement. None measures the growth of controls, rounds beyond a budget defined by findings, a guard whose verdict never varies, a victory without a negative control, or partial work destroyed at a ceiling.
Design: read, report, never block
The module is written in Python 3 with the standard library only and no network access. It installs as the package ai-watchbird-sheckley and runs as a command of the same name; audit --json emits a machine-readable report. It reads a run's event log and reports. Its exit code does not depend on the signals it reports. Its signal list is frozen, so adding a signal takes an authority token and a version change. It stays silent on a clean run, which a test enforces.
It also ships three budget rules as pure functions that a pipeline can call:
- R1,
stop_rule: one round of critique and one round of fixes per stage. A blocker counts only with a concrete scenario, a link to the declared mission goal and confirmation by the coordinator. It returnsfix_once,close,incompleteorcut, and never a new phase. - R2,
settle_at_ceiling: keeps what a participant produced before a ceiling, marked partial. - R3,
admit_control: admits a new check, round, role or gate only with an authority token; otherwise it returns a refusal as a record.
Nine signals for multi-agent oversight
The measurement part reports nine signals, W1 to W9: mandate expansion (a stage not derived from a recorded mission goal), a blind or destructive guard (one verdict on 20 or more decisions, every approval rolled back, or a required input missing), a control ratchet (controls added without an authority token), a self-feeding loop (rounds over budget, or a round opened after one with no confirmed blocker), a counterfeit victory (a win without a negative control or A/A probe), built for nobody (artifacts off by default with no callers), destroyed partial work, an unconfirmed blocker driving work, and a bound with no recorded author.
Tested against its own falsifiers
Six falsifiers state how a deployment claiming the rules would be refuted, for example if a second critique round can be scheduled without an authority token, or if stopping a process at the ceiling discards its partial output. Each has a test. A mutation script applies eighteen targeted mutations; every one turns the suite red, and a control mutation that edits only a comment leaves it green.
On six replay logs, the clean control and the morning stage produce no signals. On the reconstructed long-horizon run, where stages actually ran five, five and one rounds, stop_rule would have closed all three at the budget point.
What the module does not do
The paper is direct about the limits:
- No raw log went into the replays. They were reconstructed from published counts, and "where no exact number was published the smallest plausible value is used and marked".
- A signal reports that a pattern is present. Cause and loss are outside what a signal measures.
- It sees only what is logged. A gate never logged, a goal never written down, and work people stop attempting because a guard is there are invisible to it.
- It is digital and read-only, and must not send commands to a robot, drone, autopilot or actuator. An embodied profile is proposed as an event format only.
The module is itself a new control, so its own rule R3 applies to it first. When a port of it was built into the author's pipeline, it was admitted with a retirement condition, reported no signal on three real runs, and was removed on 15 September by a dated human decision.
Running it
Appendix B gives the commands, run from the module folder: pip install ., then ai-watchbird-sheckley audit <log.jsonl> for one event log, ai-watchbird-sheckley replay replays/ to regenerate the replay table, and python3 -m unittest -v for the unit and falsifier tests. An event log is a JSON-lines file with one event per line.
The reference implementation, tests, replay logs and the randomized test's data are archived at Zenodo as AI-Watchbird-Sheckley version 1.1.0 under the concept DOI 10.5281/zenodo.22774269, which resolves to the current version.
Full sections: the module (10), what already exists (10.1), design (10.2), falsifiers (10.3), replays (10.4), what the module does not do (10.5), inside the author's pipeline (10.6), running the module (Appendix B).
Chernets, V. (2026). AI-Watchbird (Sheckley): When Automated Oversight Widens Its Own Mandate and Harms What It Guards. SSRN 7473658. https://doi.org/10.2139/ssrn.7473658
Vadym Chernets also builds Poly A1, an app in which AIs from different companies check each other's work.