A review process for catching unchecked claimsOpen source · Apache-2.0
Not the detective's corkboard. The older sense of the word: a panel convened to tear a thing apart before it is defended for real, hostile enough that the actual review holds no surprises.
Executive summary
The problem. Documents go out with claims nobody checked. A number that disagrees with the analysis it came from. A reference that exists but credits the wrong paper. A total that changes between two pages. This happens whenever nothing sits between writing a draft and sending it — no step where someone tries to break it. People have done it for as long as they have written things down; machines now do it faster, and in better prose.
The response. A written procedure, eleven reviewer roles that each look for a different kind of fault, three scripts that enforce the parts people forget, and a fixed format for what comes back. It is free, belongs to no field, and is designed to work the same whether the reviewers are people or AI.
The method. Every sentence must either be checkable against something real — the data, the code, a source someone actually opened — or carry a visible mark saying it is not. All eleven roles run every time. The report names all eleven and says what each one found, including the ones that found nothing, because otherwise a review that ran seven of them looks exactly like a review that ran all eleven.
The cost. Eleven reviewers read the draft. One person decides which findings to act on and makes the fixes. Then the corrected document is read again by someone who has not seen the findings, because fixes break things — up to three such rounds, then it stops whether or not it has converged. A caption or a one-line summary does not get eleven reviewers; one person walks the same eleven checklists in a single pass — except that the citation role stays a separate reviewer whenever the document attributes a method or claims novelty.
The limits. A clean report proves the reviewers did their jobs. It does not prove the document is right, and whether this process finds more than some other approach has never been measured. What it does not do is the section to read before adopting it.
Everything down to What you hand over, and all of What it does not do, assumes no technical background. The gates and Implementation assume you work with code, and define their terms as they go.
Why it exists
Slop is not bad writing. It is confident writing that nobody checked. It reads as finished. Nothing on the page separates a verified sentence from a plausible one.
People have always done this. Citations copied from other citations until the original says something else. A number transcribed wrong once and repeated for a decade. A methods section describing what was meant to happen. Machines repeat these faults faster and in better grammar; the faults are the same ones. The process does not care who wrote the draft.
Some documents are assembled by a program — a chart drawn from a data file, a report generated from a template. There, a number can match its caption exactly and still be wrong, because the program that produced it used a statistical routine incorrectly, or re-derived something the project already had working.
A figure is only as sound as the method behind it, so the review reads the code that made the number. Two seats on the panel exist for that.
An appendix to the process document records an example incident behind most of the rules. They are internal to the project that produced them, so a reader cannot audit them from outside — which makes them an account of where the rules came from, not evidence that the rules work. ⚠
The core principle
Every sentence must be either verifiable against a real source — the data, the code, a prior result, a checked citation — or explicitly flagged as unverified.
No unsourced claim. No invented citation. No contradiction. No filler. What cannot be verified is neither deleted nor quietly kept: it ships marked ⚠, so the reader can see which sentences are not backed.
The process
Six steps, and one of them loops. The order is load-bearing: step 0 runs before the draft exists, and step 4 is both the step people skip and the step that catches the fixes that broke something else.
You do not install this; you copy the files into your own project. So your copy drifts behind the original, and reviewing against a stale copy skips rules you already paid for. A script checks this — see the gates.
Write it as you always would. The point is that this draft is not what you deliver.
Every role runs, every time. What changes with the stakes is how you run them, not which. A long document gets eleven reviewers. A one-line caption gets one person walking all eleven checklists — with one exception. Any document that attributes a method, or claims something is novel, unattributed, or its own, runs the citation role as a separate reviewer however short it is. A role that finds nothing says so, and says what it checked: silence and absence must not look alike.
Collect the findings, drop the duplicates, rank them, and decide each one: fix it, flag it, or reject it with a reason. Then make the fixes. Note who is deciding — usually the person who wrote the draft, which is a conflict worth naming and recording.
Two passes, in this order. The blind pass re-runs the roles against the repaired document with no knowledge of the earlier findings, the fixes, or which parts were touched — because a reviewer told “we fixed the caption on page 12” checks page 12, confirms it, and never sees what the fix broke elsewhere. Only then does the follow-up pass walk the original list and rule on each finding: fixed, not fixed, moved, or superseded. Moved is the verdict this pass exists to produce.
It stops on severity, not on silence. Stop when a blind round produces no blocking and no major findings, or after three rounds, whichever comes first. If severity is not falling across rounds, stop and escalate to a person — a flat or rising count of serious findings means the document has a structural problem that patching will not retire. Minor findings surviving the last round are recorded as residual ⚠, not fixed. If a document is assembled by a program, rebuild it and check the rebuilt file — the program is not the deliverable. Make the last action a rebuild, never a fix.
The corrected document, a plain-language summary, and a role ledger. See what you hand over.
The review team
Each role gets the draft and the real sources — the data, the code, the companion documents — and returns findings in one shape: location, issue, severity, suggested fix, and whether it could be checked against a source.
The division between roles is not obvious, and the reasoning behind it is the part worth copying.
“Prove It”
Claim & data verifier
Pulls every factual and numerical claim and checks each against the data, the code, or the earlier result. Returns a table: quoted value, cited source, recomputed value, and one of match, mismatch, or unverifiable. It recomputes rather than eyeballs.
“DOI or Die”
Citation & reference validator
Confirms each reference exists and says what is quoted. Then the half that gets skipped: is it the origin, or merely the earliest source the reviewer reached? Follows the cited work's own references back until they stop, and reports where it stopped. A shared author is not a shared laboratory.
Trace forward too. Going backwards finds where a method came from; it does not find what its authors did with it next, which is usually where the closest prior art for your use of it lives. And ask what the people around you already know: an email to a tool's author, or an enquiry answered months ago and never written down, is real evidence, and it is invisible to every literature search that will ever be run.
No guessed bibliographic detail, ever. This is the one role the size rule may not collapse (the trigger is in step 02) — one pass inherits the writer's search history and stops where it stopped.
“Cross-Examiner”
Consistency auditor
Checks counts, totals, terms, and whether the figures and the text agree — within the document and against its companions. Watches for one group counted two ways, which is how the same total changes between sections without anyone noticing.
“Reviewer 2”
Adversarial reviewer
Reads as a hostile peer reviewer: overreach, unsupported leaps, missing caveats, undefined quantities. Asks the question that kills a soft result — could it ever have come out otherwise?
“Kill Your Darlings”
Line editor
Cuts every sentence that has not earned its place. Hunts undefined jargon, ambiguity, redundancy, and broken order. Each sentence must assert exactly one true thing.
“RTFM”
Methods / domain expert
Reads the source paper and the tool's own documentation before reviewing, whenever the document rests on a particular method, model, or piece of software, then checks the work actually obeys that method. Never reasons from memory about what a tool does.
“Reinventing the Wheel”
Reuse auditor
Catches new code redoing what the project already does in code that is tested and working — and, where it does redo it, whether it matches the original in every detail: the same settings, the same units, the same checks for bad input.
“You Lost Me”
Naive-reader accessibility
Reads with no prior knowledge and marks every place a cold reader is lost. A document can be right in every number, honest, and cleanly made, and still be unreadable to the people it was written for.
“Show, Don't Tell”
Density & figure-first
Asks what no other role asks: what here should have been a picture? Drafts default to prose — correct, complete, sourced, and unreadable at a glance.
“Ship It”
Build & craft gate
Owns every check settled by opening the finished file as the reader will see it, or by running a script: text running over a figure, axes with no labels, words cut off at an edge. It answers in a table, not prose, so a skipped check leaves a visible hole.
“Start With the Problem”
Argument order
Reads only the order. A document can be true, readable, and clean on every page and still fail, because it gives the fix before the reader knows there is a problem.
Why the roles divide this way. Two reasons, both from failures.
What it costs to answer. You can satisfy a judgment call (“would a cold reader follow this?”) by thinking. You can satisfy a mechanical check (“is this axis labeled?”) only by opening the file. Give one reviewer both, and the prose answer covers for the file nobody opened. So every mechanical check sits alone, in seat 10.
What gets read at once. Most roles read one page at a time. A fault that belongs to the whole sequence, or to the whole page, is invisible to all of them: each page passes on its own. Seats 9 and 11 read the whole sequence and the whole page, so the reader is not the one who finds them.
A role that looks inapplicable is read by its checklist, not its title. The role that judges itself out is the one that would have caught it.
Seat 5's nickname is the popular corruption of the line. Arthur Quiller-Couch wrote murder your darlings, not kill, in On the Art of Writing (1916) — the older word, and the better fit here.
The output contract
Not the draft plus a list of its faults. The repaired document — and if a program builds it, the rebuilt file, newer than the last fix and newer than everything it draws on.
What was checked, what was found and fixed, and every ⚠ still standing. Plus a table of findings by severity for each round — blocking findings running 6, 0, 3, 0 shows a review converging in the way that matters — and the reason it stopped: severity floor reached, or round cap reached.
One row per role, all of them, each carrying its findings or its “no findings, and here is what I checked” line. If the panel found nothing, say so. Never invent findings to look thorough.
The gates
A gate is a check placed in the path of the work, so it fires whether or not anyone remembers it — the way a smoke alarm is not a rule about smoke. Two rules of this review process were prose, and a third governs how parallel AI sessions talk to each other rather than how documents are reviewed. Each was skipped exactly when it mattered, and each is now a script that runs by itself and stays quiet when the answer is fine. Take this part even if you take nothing else. It works on any rule you cannot afford to leave to memory.
Four words this section uses
0 means fine; each other number
means something specific. It is how one script's verdict can drive another's behaviour.| The question it answers | How | What it reports |
|---|---|---|
Is your copy current?murderboard_ |
Compares your copy's stamp against the original. Runs quietly at the start of a
session, reading a cached answer; runs again, checking directly, at the moment of review.
Point it at any project you copy files from with --label, --slug
and --file — nothing about the mechanism is specific to this
process, though its defaults are. |
0current 1stale 2could not tell |
Does the report account for every role?murderboard_ |
Reads the list of roles out of the process document, never from memory, and checks the finished report accounts for all of them. Add a role upstream and every copy's check picks it up as soon as that copy is refreshed — with no script edit anywhere. |
0all present 1a role is missing 2could not tell |
Does the thing you are about to describe exist yet?require_commit_ |
For setups where several AI sessions work in parallel: refuses to send a message between sessions while anything in the project is uncommitted. It cannot read the message and does not try — it gates on there being a commit at all. Nothing stores these messages, so what they describe must already be saved. |
0allow 2block |
0 pass1 the thing it checks is wrong2 the check itself could not run
Where the roster gate came from. “Every role runs” was prose. Then a run using 7 of 11 roles and a run using all 11 produced reports no reader could tell apart. “No findings from role 9” is worth nothing if role 9 never ran. So the ledger became required, and then a script began checking it.
What that gate does and does not prove. It reads the report, not the run — an eleven-row ledger written by a reviewer who ran seven roles passes it. What it buys is that a silent omission becomes a written falsehood, which is a real raise in cost and is not the same as proof. ⚠
Two of the three return “could not tell” rather than a false “fine” — with one deliberate exception. The freshness gate in session-start mode serves the previous answer from a cache, so a copy that went stale since the last session is reported current, one session late. That is why the same gate runs again, checking directly, at the moment of review. The third gate has no “could not tell” verdict at all and allows when it cannot see a project.
Every gate ships --selftest, which proves each branch can still fire.
Read this before you quote a run
A clean run is evidence the roles ran. It is not evidence the document is correct. The process requires you to say so, in the delivered summary, in these terms or equivalent:
This review found and fixed N defects. It is not a correctness proof. The round-by-round table measures how quickly reviewers stopped finding things, not whether anything remains.
The run record is the most quotable thing the process produces, and 11 of 11 roles, nothing left above the floor reads to anyone as a clean bill of health. The role ledger fixed one confusion and left the next one standing: “11 of 11 and clean” and “11 of 11 and correct” also look alike, and the second is what a reader takes away. Do not let a clean report stand in for someone competent having read the thing.
Two claims are separable, and only one of them is established. Mechanically you can check today that a given report names every role, that a copy is stale, and that the review loop terminates by construction. The project's automated tests cover the first two. Empirically — that this finds more, or better, than some other approach — nothing here has been measured against a baseline, and no rate is claimed. ⚠
It also cannot see its own misses. A process observes the defects it catches and never the ones it does not, so its miss rate is unknown and not knowable from inside.
The panel checks that claims are sourced, consistent, traced to their origin, and legible. It cannot supply judgment nobody involved has. Someone who knows the field will catch, half asleep, what this misses at full effort: that a result is implausible for reasons no source states, that the method is sound but wrong for the question, that the interesting finding is the one nobody wrote down.
Run it as a floor under expert review, never as a substitute.
Reviewers drawn from one model, given one draft and one house style, share their blind spots by construction. The eleven seats buy coverage of angles, not statistical independence, and no table can tell a document with nothing left to find from one whose reviewers all looked in the same wrong place. A human on the panel is the only real decorrelation available.
Every rule here was added after something slipped through. That is the only evidence a rule is worth having — and it is evidence the defect exists, not that the rule catches the next one. The set covers the mistakes already made and says nothing about the next. Expect to add to it.
What review this page has had, since it argues that documents should say.
Drafted and self-reviewed, then read once by a second reviewer at an earlier version, whose three findings were applied. Most of the current wording post-dates that pass. On 2026-08-25 the full panel ran against this page for the first time — all eleven roles, then a blind re-review by a reviewer shown neither the findings nor the fixes. Among what they caught: a false claim that the page was self-contained, a fabricated “prose for years”, one rule stated three contradictory ways, an output contract naming an artifact the process had retired, muted text failing contrast in both themes, and a label in the diagram above that was painted underneath the boxes and asserted the wrong stopping rule.
The run record — every role's ledger row, the round-by-round table, the
stopping reason, and the residual ⚠ flags — is committed
beside this page at docs/reviews/explainer_murderboard_2026-08-25.md.
Read it rather than this paragraph: it is the auditable one. This page is an honest example of
its own subject — reviewed enough to be worth reading, not enough to be quoted as
verified.
Implementation
Nothing to install and nothing to subscribe to. Each project keeps its own copy of the files, so anyone who clones your project gets a working murderboard with it. Copies go out of date in silence, which is why the freshness gate exists.
To try it in two minutes with no setup at all, open PROMPT.md, copy the block, paste it into any AI chat, then paste your document. Its role list is generated from the process document, so it cannot drift. START-HERE.md is the same idea at more length. The steps below are for wiring it into a project permanently.
Three more words the steps below use
The process document goes under docs/; the two review gates, the re-vendor
tool and the literature tool under tools/; the message gate to
.claude/hooks/, where its own wiring looks for it; and — if you use Claude
Code — the skill to .claude/skills/murderboard/. Each copy carries a
one-line stamp naming the version it came from, so drift shows.
# once, to describe what your project vendors python3 tools/murderboard_revendor.py --example-config > .murderboard-vendor.json # thereafter, to re-copy and bump the stamps surgically python3 tools/murderboard_revendor.py
Use that tool rather than a search-and-replace over the files: the obvious one-line
sed rewrites every stamp-shaped string in a file's body, and the freshness gate
contains eleven of them.
Run freshness at the start of a session, as an early warning. Run the roster check against every finished report — in your CI, if you want it to block rather than merely report.
# session start — silent unless your copy is stale bash tools/murderboard_freshness.sh --hook # after a review run — exit 1 if a role left no trace bash tools/murderboard_roster.sh check REPORT.md
Reviewers need real papers, not remembered ones. fetch_paper.py fetches only
from open-access hosts, keeps what it gets, and adds anything paywalled to a list for a human
rather than scraping it. Searching your own library is a step you run first, with
--have; promoting a keeper into that library is what makes the next search find
it. It needs Python 3, plus pypdf or pdftotext to read PDFs.
export MURDERBOARD_LIT="/path/to/your/pdf/library" python3 tools/fetch_paper.py --have smith attention # check the library first python3 tools/fetch_paper.py <url> # fetch, keep, print text python3 tools/fetch_paper.py --promote <url> --name "Name.pdf" # file a keeper python3 tools/fetch_paper.py --need "<citation>" # flag one you can't reach
Add a rule to your project's instructions — CLAUDE.md, a contributing
guide, a team norm — that documents go through the murderboard before delivery. The repo
ships a paragraph to paste. Skip this and the files sit unused. The steps above make the rule
enforceable; this one states it.
Install the skill and run /murderboard <document>. It runs the parts that
must not depend on memory: it checks freshness at the moment of review, not just at startup;
reads the role list from the process document; resolves the document to the built file
rather than the program that generates it; records a checksum before and after; and writes a
record the roster gate then checks.
None of this requires AI, and none of it requires the scripts. The eleven roles are a checklist and the gates are an optimisation. Run the panel as human review — one person in eleven passes, or eleven people in one — and hand back the same three things. The structure and the record do the work, not the reviewer's identity.