Decomposed-criteria scorer

Ask a model to rate a candidate for overall fit and it will produce a number that feels reasonable and cannot be argued with. Decomposition breaks that habit. Each candidate leaves triage with one score per sub-criterion, each score carrying the evidence it was drawn from, and the combined rank is computed afterward by an arithmetic rule a person wrote.

The reason to work this way is that disagreement becomes locatable. Two analysts who both dislike a candidate may dislike it for opposite reasons, and a single fit rating hides that completely. Four separate cells show which one they actually differ on, and that cell is the thing worth discussing.

This step is expensive. It multiplies model calls by the number of sub-criteria, and it needs a scale and an evidence definition for each one. Run it on a pool that has already been cut, after the rules-based rejection of candidates failing hard criteria has taken out everything ineligible on its face. Decomposed scoring on an uncut pool is the most reliable way to spend a research budget on records that were never eligible.

What comes out is still a triage artifact. A row of sub-scores says which candidates deserve a closer read. It does not say which one is right, and it is not an assessment.

A sub-criterion needs three things defined before the first call: what it is asking, what evidence answers it, and what each point on the scale looks like. Skip the third and the model will invent anchors. A four-point scale with written anchors produces scores two people can check against each other; a one-to-ten scale with no anchors produces a distribution clustered on seven.

Name the evidence source per criterion, not per candidate. If a criterion asks about technical maturity, say that the answer comes from the publication and patent record and nothing else. If a criterion asks about team depth, say that it comes from the founder and staff listing. Criteria that read the same input twice are not independent, and the pipeline will report agreement that is really an echo.

The criteria themselves come from scope. Sub-criteria are a decomposition of what the hypothesis tree and screening criteria already specified, not a new list assembled by whoever wrote the prompt. When decomposition reveals that the original criteria overlap, take that back to the scope owner. A hidden overlap becomes a double weight later.

Every scale needs an off-scale value for evidence that is absent. Not "one" and not the midpoint. A candidate with no public information about manufacturing readiness has an unknown, and unknown has to survive into the rank rather than being quietly converted into a low score.

Run each criterion as its own pass over the pool. The prompt for a pass names one criterion, its anchors, and the evidence field it may read. It does not mention the other criteria, the candidate's other scores, or the eventual ranking. Independence is a property of how the calls are made, not something you can ask for in an instruction.

Passes that share a context window leak. A model that has just rated a candidate strong on scientific novelty carries that assessment into the next question, and the commercial score drifts up to match. Reset between criteria. If the platform reuses conversation state, that reuse is a defect in the pipeline, not a convenience.

Every score returns with the span it rests on. A cell showing three and a quoted sentence from a patent abstract can be checked. A cell showing three alone is an assertion with a number attached. When the model cannot produce a span, the cell returns unknown rather than a guessed value.

Repeat each pass more than once where the criterion is subjective. The spread across repeats belongs in the output next to the score, because a criterion that returns two, four, and three across runs is telling you the anchors are too loose. That spread is also the input for surfacing candidates the raters split on.

The scenario below is invented, including the counts and the company names. A corporate battery research group is scouting external partners for solid-state electrolyte work. Their sourcing sweep and knockout pass leave 88 organizations: university groups, national lab spinouts, and venture-backed startups.

Scope defined four sub-criteria. Materials fit, meaning the chemistry family matches the platform the group is committed to. Demonstrated cell performance, meaning cycle and rate data exists in a peer-reviewed or third-party-verified form. Manufacturing path, meaning a stated route from lab coupon to pouch cell. Partnership availability, meaning the organization is not exclusively tied to a competing automaker.

Each criterion has its own evidence source and a four-point scale with written anchors. Materials fit reads publications and patents. Cell performance reads reported data only. Manufacturing path reads company statements and pilot line disclosures. Availability reads announced partnerships and licensing notices.

One candidate, a lab spinout called Kirreth, scores four on materials fit, four on cell performance, unknown on manufacturing path, and two on availability. Another, a startup called Novadyn, scores two, two, four, and four. A single blended rating would have put both near the middle and told the scouting lead nothing. The decomposed rows say something usable: Kirreth is a science bet with a possible exclusivity problem, Novadyn is a process bet with thin data.

The published weighting, written during scope, is materials fit at 0.35, cell performance at 0.35, manufacturing path at 0.2, and availability at 0.1, with unknowns excluded from the denominator and printed on the row. Twelve candidates carry an unknown in at least one cell. Those twelve go to the closer read first, because a missing cell is a question, not a weakness.

Decomposition fails in specific ways, and all of them end with a single opinion wearing four coats.

Criteria that read the same evidence are the first. "Scientific strength" and "publication quality" scored from the same paper set will move together, and the recombined score counts one judgment twice. Check correlation across the pool after the first round. Two criteria that never disagree are one criterion.

Weights chosen after the scores are visible are the second. Someone looks at the ranking, dislikes the top entry, and adjusts a weight until the order looks right. That is a judgment call presented as arithmetic. Publish weights before the pass runs, and log any change with a reason and a date. A tracker comparing rater scores against how the calls turned out is the honest way to revise them, not a quiet edit during a meeting.

Unknowns silently becoming zeros is the third, and it is the one that changes the ordering most. A candidate with no public manufacturing statement is not a candidate with a bad manufacturing plan. Decide the missing-data rule in advance, write it on the artifact, and show which cells were affected.

The fourth is scale drift across a long pool. Anchors mean less by the eighth hundred candidate than they did at the tenth. Seed each pass with a few pre-scored reference candidates and check that they still land where the team put them.

The recombined rank is a starting order, not a decision. The scouting lead reads the rows, not the totals, and moves candidates up or down with a written reason. That reason is the useful record. Six months on, the interesting question is which cell someone overrode and whether they were right.

Unknown cells belong to a person too. Someone decides whether to look for the missing evidence, drop the candidate, or advance it with the gap declared. Letting the arithmetic resolve unknowns removes exactly the judgment this method was built to expose.

Nothing here should be read as an assessment. Sub-scores are compressed public signals, gathered fast and cheaply. The candidates that survive go into a fuller read, where a structured one-pager compiled per candidate can put the same claims in front of someone who checks them against the sources.

Keep the scored table as it was produced. Rounds get rerun, criteria get revised, and the version that shaped the shortlist is the one that has to be recoverable. A table overwritten in place cannot explain why a candidate was cut.