Multi-agent adversarial review

Adversarial review is not a second opinion. It is a set of structured attempts to break a draft that already exists. The draft has to be finished enough to attack. Running critics against an outline produces criticism of an outline.

The output is an objection list, and each objection has four fields. What claim it targets. Why the claim might be wrong. What evidence would settle it. And how much the conclusion moves if the objection holds. An objection missing the fourth field is noise.

Depth is a configuration choice, not a different system. A light pass runs two personas over the executive claims. A heavy pass runs six personas, a falsification sweep, and a quantitative chain over every load-bearing number. The cost difference is large and the plumbing is identical.

Decide the depth from what the decision can tolerate. A reversible pilot does not need six reviewers. A commitment that runs for four years and cannot be unwound does. Matching effort to consequence is the only thing that makes this affordable.

Telling a model to be skeptical produces skeptical-sounding prose about the same points. Real heterogeneity comes from what each reviewer is given and what each is asked to do. Change the corpus, the question, and the success condition, not just the tone word.

A methodologist reviewer gets the methods section and the declared protocol. Its question is whether the design can support the claim, and it may ignore the topic entirely. A domain reviewer gets the field literature and the draft's substantive claims. Its question is whether anything here contradicts what practitioners already know.

A base-rate reviewer gets historical outcomes for comparable efforts, and nothing about this one's merits. Its only question is how often things of this shape have worked before. That reviewer produces the most uncomfortable objections, because it is immune to the narrative.

Add a stakeholder reviewer when the work will land on someone. Give it the perspective of the group affected, and ask what the draft assumes about their behavior. Programs fail on those assumptions more often than on their arithmetic.

Write each persona's brief as a separate configuration file, versioned. Reviewers that drift between rounds make objection volumes incomparable. The same discipline applies here as in re-scoring dropped candidates with an advocate persona, where the persona's brief is the instrument.

Falsification runs before persona review, and it is the cheapest useful step. Extract each load-bearing claim from the draft as a standalone sentence. For each one, ask what observation would prove it false. Then ask whether that observation was looked for.

Most drafts fail this immediately on two or three claims. The claim turns out to be constructed so that nothing could disconfirm it. That is a finding about the draft, not about the world. Rewrite the claim until it is falsifiable, or drop it.

The three-role chain handles quantitative claims. A hypothesis agent states the claim precisely, including population, period, and units. A verifier agent checks whether the cited sources actually contain that claim. A quantifier agent checks whether the arithmetic and the aggregation hold.

Keeping them separate matters more than it looks. One agent asked to do all three will resolve its own tensions silently. Three agents produce three records, and the disagreements between them are visible. A verifier passing a claim the quantifier fails is the most informative result the chain produces.

The quantifier stops at the arithmetic in the draft. Testing whether conclusions survive different parameter choices needs the actual model. That is running alternative parameterisations against the real code, which is a heavier and more decisive step.

The following is hypothetical, with invented findings. An impact investment team has drafted a deep dive on index-based drought insurance for smallholder farmers. The draft recommends a three-country expansion. Halima Sesay owns the deep dive and will present it.

Four reviewers are configured for this pass. A methodologist and a domain reviewer with agricultural insurance literature. A base-rate reviewer holding outcomes from prior index products. And a farmer-perspective reviewer. Falsification runs first, across eleven load-bearing claims.

Falsification kills one claim outright. The draft states that basis risk is manageable, without saying what unmanageable would look like. No threshold, no measurement, nothing that could come back false. The claim is rewritten with a payout-mismatch threshold before review continues.

The four reviewers return thirty-one objections between them, and overlap is heavy. After merging, nineteen distinct objections remain. Only five carry a stated conclusion impact large enough to change the recommendation. Those five are the meeting.

The base-rate reviewer produces the sharpest one. It notes that prior index products in comparable settings lost enrollment after the first non-payout year. The draft models four years of stable uptake and never addresses the second year. The farmer-perspective reviewer arrives at the same place from a different direction.

Halima accepts three objections, rejects one with a written reason, and sends one for testing. The recommendation survives, narrowed to two countries and one year.

The commonest outcome is too many objections and no way to sort them. Six reviewers over a long draft will return more than anyone reads. Require an impact estimate on every objection and sort by it. An objection that cannot state what changes if it is right goes to the bottom.

Duplicate objections in different vocabularies inflate the count. Three reviewers noticing the same weak assumption is one objection with three sources. Merge before the list is presented, and record how many reviewers raised each merged item. Independent agreement is worth knowing about.

Agreeableness is the quiet failure. Models asked to critique a well-written draft will find small, safe problems. The prose is strong, so the objections come back about phrasing and scope caveats. Retire a reviewer configuration whose objections are never accepted.

Adversarial review can also be used as decoration. A team that runs the pass, records nineteen objections, and changes nothing has bought a paragraph. The paragraph says the work was rigorously reviewed. Track acceptance rate over rounds; a rate near zero means the pass is theater.

Format compliance is not the same as substantive attack. A draft can satisfy a protocol and completeness check and still rest on a claim nobody tried to break. Run both, and do not let either stand in for the other.

Objections do not resolve themselves. A named person goes through the merged list and marks each one. Accepted and the draft changes. Rejected with a reason. Or open, with a test attached. Nothing is left unmarked.

Rejection is a legitimate ruling and it should be common. Many objections are correct in general and irrelevant to this decision. Write the reason anyway, because that sentence is what a later reader will need. "Considered and rejected" is not a reason.

The reviewers hold no authority over the recommendation. They produce candidate weaknesses; the owner decides which ones matter. A workflow where an objection automatically downgrades a candidate has handed judgment to a configuration file. Configuration files do not stand behind decisions in a meeting.

Open objections need somewhere to go. Each one is a question with a stated resolution test. That is most of a research brief already. Send them to an unresolved uncertainty packaged for a named consumer rather than letting them expire in a review file.

Carry the accepted objections into the decision meeting rather than summarizing them away. The strongest surviving objection belongs in the case against each finalist circulated beforehand. A committee that never sees the objections is deciding on a cleaned-up draft. The point of the whole pass was to stop that from happening.