Code-grounded quant red-team

A deep-dive often rests on one number that came out of a model nobody on the team has run. A code-grounded red team narrows that discomfort into something checkable. It returns a table in which each row names one assumption, the value the authors used, the alternative value tested, the source for that alternative, and what happened to the headline conclusion when the model was executed again.

The table does not declare the model wrong. It shows where the conclusion is load-bearing on a single choice and where it is not. A result that survives every defensible parameterization is stronger than the original paper claimed for it. A result that flips at a value other researchers routinely use is fragile, and fragile is not the same as false.

Keep this apart from reading work. Checking that each cited passage actually supports the claim built on it operates on text. This operates on execution. A citation check can confirm the paper says what the brief says it says. Only a re-run tells you whether that sentence survives a different discount rate.

Three artifacts have to be in hand: the code, the data it reads, and the authors' own statement of which parameter values they chose. Add the environment, because language version, package versions, and the random seed decide whether two runs of the same script agree at all.

Reproduce the baseline first. Run the unmodified package and compare the output against the published figure. If the two do not match, stop. A reproduction gap is a finding by itself, and it makes every downstream flip uninterpretable, because nobody can tell whether the conclusion moved from the change you made or from the mismatch you never explained.

When there is no runnable code, say so and stop there. A narrative about sensitivity, written by a model that executed nothing, is exactly the artifact this method exists to replace. Record the model as unverifiable, and treat the claim as resting on the authors' word rather than on evidence the team inspected.

Which parameters deserve the compute is a scoping question, and ranking inputs by how much they would reduce decision uncertainty is where that ranking already lives. Testing forty parameters that cannot move the decision is a slow way to produce a long appendix.

Not every parameter deserves the same treatment, and mixing the three kinds is how a red team produces noise instead of findings.

Contested parameters have defensible alternatives in the published literature: a discount rate, an elasticity, a degradation curve, a time horizon. Test these against values you can cite, and put the citation in the row. An alternative with no source behind it is a preference, not a test.

Arbitrary parameters were chosen for convenience: a truncation point, a winsorizing threshold, a rule for dropping outliers. These deserve a range rather than one rival value, because there is no correct answer for the authors to defend.

Structural assumptions are the functional form rather than a number: linear where the mechanism may saturate, additive where effects may interact, independent where they clearly are not. Changing these rewrites the model, so label those runs as an alternative specification instead of a sensitivity test.

Run one assumption at a time first, then run the combinations that plausibly occur together. Report the flip point wherever you can locate it, meaning the value at which the conclusion changes sign or crosses the threshold the decision actually uses. A flip point is more useful to a reviewer than a percentage change, because a reviewer can judge whether that value is plausible in the field.

Nothing in the following case is real; the model and its inputs are constructed for the example. An infrastructure research team is assessing a vendor-published model claiming that a four-hour grid battery installation pays back inside eight years at a named substation site. The model ships as a spreadsheet plus a short Python script, reading a public tariff file.

Baseline reproduction works. The script returns a payback of 7.6 years, matching the published number closely enough to proceed. The declared parameters include a 4 percent discount rate, annual capacity fade of 2 percent, 340 full cycles a year, and a tariff schedule held flat after year five.

Four alternatives get tested, each with a source attached. The discount rate moves to 7 percent, the figure the team's own capital guidance uses. Capacity fade moves to 3 percent, the upper end of the manufacturer's warranty language. Annual cycles drop to 260, matching the dispatch history of a comparable site. The flat tariff assumption is replaced by the regulator's published escalation path.

Two of the four flip the conclusion. At 7 percent, payback moves past the eight-year bar to 9.4 years. At 260 cycles, it moves to 10.1 years. Fade and tariff escalation each shift the answer by less than half a year and leave the claim standing. Run together, the discount rate and the cycle count push payback beyond the asset's own stated warranty period.

The finding written into the deep-dive is not that the vendor model is wrong. It is that the payback claim holds only at the vendor's discount rate and at a cycle count the comparable site has never reached, and that the team's standard capital guidance breaks it without any help from the other three tests.

The most embarrassing failure is a flip produced by the re-run rather than by the assumption. Someone changed a units column, broke a join, or edited a parameter the script reads twice from two different places. Diff every alternative run against the baseline and confirm that only the intended cell moved.

Flip-hunting is the next problem. Given enough parameters and enough freedom on their ranges, almost any conclusion can be broken by someone determined to break it. If the alternative value sits outside what any published source uses, the flip was manufactured. Bound every range with something citable, and record the bound on the row.

Scripted and versioned runs are not optional here. Manual edits to a spreadsheet leave no trace of what changed, and three weeks later nobody can say which version produced the number that reached the memo. Keep each run as code with a parameter file and a run identifier a colleague can re-execute.

Tone is the quieter failure. A fragility table reads like an accusation when it travels without context, and authors who feel accused stop sharing replication packages with the field. Report what flipped, at what value, from which source, and leave the adjectives out.

Fragility is not itself a conclusion. Someone has to say whether a payback that holds only at the vendor's discount rate is disqualifying, tolerable with a stated condition, or irrelevant because the decision never rested on it. That judgment belongs to the person whose name goes on the assessment, and it should be written down next to the table.

Carry the fragility forward instead of collapsing it. A conclusion that survives two of four alternatives should reach the decision meeting as a range with its flip points attached, which is the shape decisions recorded as probability distributions already expects. A single point estimate with a footnote loses the finding on the way.

Assumptions the red team could not test belong in the open pile rather than the appendix. A parameter with no citable alternative, a model with no runnable code, or a structural choice nobody on the team can evaluate is work for a specialist, and packaging each unresolved uncertainty as its own research brief is where those items should land.

Where a flip turns on a judgment rather than a number, no re-run will settle it. Send that one to structured adversarial review with heterogeneous personas, where the argument is the object of study rather than the code.