Outcome-linked prior-art retrieval

Search over an internal archive returns documents. That is useful and it is not what this step is for. The output here is a set of prior cases where each one carries three things: what the team recommended, what the organization decided, and what happened afterward. A recommendation with no recorded outcome is a document. A recommendation with an outcome is evidence.

Analysts want this for a specific reason. When a new candidate resembles something the organization looked at four years ago, the question is not what the memo said. It is whether the memo was right. Teams that retrieve only the document repeat the reasoning of colleagues who turned out to be wrong, with fresh confidence.

This is the highest-effort item in the source stage and the effort is not in the retrieval. Embedding an archive is a weekend. Joining each memo to the decision it fed, and each decision to a recorded result, means touching systems that were never built to talk to each other. Most organizations discover mid-build that the outcome data does not exist in a usable form.

Discovering that is itself worth something. An organization that cannot say what happened after its own decisions has a problem larger than search.

Three joins are needed and each one breaks differently. Memo to decision: the analysis document has to be tied to the committee item it supported, which usually means matching on a case reference that changed format twice. Decision to implementation: what was approved is often not what was built. Implementation to result: this join requires a monitoring or evaluation record that may only exist for a subset.

Define what counts as an outcome before building anything. For a funder it might be whether the funded work produced what it promised. For an investment team it might be a return at a fixed horizon. For a policy unit it might be whether the measure was implemented at all. Pick one definition per case type and record which definition each row uses.

Set an outcome horizon and stick to it. A case decided eighteen months ago has no outcome yet, and it must not be scored as neutral or successful because nothing bad has been recorded. Mark it pending. Pending is a real state and it should be visible in every result set.

Where the join cannot be made, return the document with an empty outcome field and say why. No evaluation was commissioned differs from evaluation exists but is not accessible, and both differ from the case was withdrawn before implementation.

Collapsing these three into a summary destroys the most useful pattern in the archive: cases where the team recommended one thing and the committee did another. Those are the rows a new analyst should read first, because they show where the organization's stated criteria and its actual bar diverge. Rows like that exist only if someone captured the decision and the disagreement behind it.

Add two more fields that cost little and matter a lot. The decision date, because the environment moves. And the names on the memo, because a reader who can call the author gets more in ten minutes than the retrieval system ever will.

Do not let the system score the past case as a success or a failure. Return the recorded result and let the reader judge. A project that met its targets in a context that no longer exists is not a success story for the current candidate, and a compressed verdict hides exactly the detail that decides whether the analogy holds.

Retrieval should also respect access. Internal archives contain material that not everyone may read, including commercially confidential appraisals and personnel-adjacent notes. Scope the index to the requesting team's permissions rather than filtering after retrieval. This is where the pattern differs from surfacing the implicit bar in past decisions, which reads decisions in aggregate rather than pulling individual files.

The case below is made up. A development bank's investment research team is appraising a smallholder irrigation scheme in a river basin the bank has financed in before. The analyst wants everything comparable the bank has appraised, with what happened to each.

Retrieval returns twenty-three prior appraisals over fifteen years. Eleven have a completion report. Four have an independent evaluation. Six have a decision record but nothing after it. Two are within the outcome horizon and marked pending. The analyst is now looking at a much smaller real evidence base than twenty-three documents suggested.

The four with independent evaluations carry the most weight. Three of those four record the same thing: the scheme was built, the water arrived, and the water user association that was supposed to maintain the canals did not function after the project team left. In two of the three, the original appraisal treated maintenance capacity as an assumption rather than a risk.

The recommendation and decision columns add something the documents alone would not. On two of the twenty-three, the appraisal team recommended against and the board approved anyway, both times citing regional commitments. One of those two is among the three maintenance failures. That is not a finding about causation. It is a reason to read those two files carefully.

The analyst adds a maintenance capacity criterion to the current appraisal, with a specific test drawn from what the evaluations described. The six cases with no outcome record go on a list for the evaluation department, because the gap is worth fixing before the next basin.

The archive is not a sample of decisions. It is a sample of decisions that produced documents, and those skew toward cases that went far enough to generate paperwork. Candidates rejected early leave a line in a log and nothing else. Any pattern read off the retrieved set is a pattern about the cases that survived to be written up.

Outcome records skew harder. Evaluations get commissioned for large cases, contested cases, and cases somebody wanted defended. Quiet successes and quiet failures both go unrecorded. Show the coverage rate alongside every result set: how many retrieved cases have an outcome, and of what kind. Without that line, eleven completion reports look like eleven observations of the population.

The third hazard is the persuasive old memo. Internal documents were written to win an argument, and a well-argued appraisal from a respected colleague is disproportionately convincing years later. Read the outcome field before the document. Reading in the other order means the memo frames the outcome instead of the outcome testing the memo.

The fourth is quiet duplication. The same case appears as a concept note, an appraisal, a revised appraisal, and a board paper, and a naive retrieval returns four rows that look like four precedents. Resolve them to one case before counting anything.

Similarity in an embedding space is not comparability. Two irrigation schemes can read almost identically and differ on the thing that decided the outcome: the tenure arrangement, the rainfall regime, the capacity of the implementing agency. The analyst names which dimensions have to match for the precedent to count, and that judgment does not belong to the retrieval system.

Cite prior cases the way external sources are cited. Case reference, date, what was decided, what the outcome record says, and where to find it. A current memo that says the bank has seen this before, without naming the cases, is an appeal to institutional memory rather than evidence.

Feed corrections back. When an analyst finds that an outcome field is wrong, or that two records are the same case, that fix has to reach the index. An archive that degrades quietly is worse than no archive, because people trust it for a few more years before they stop. The structured check-in loop with recipients is where much of this correction originates.

And keep the retrieved set in the current round's file. Someone will later ask which precedents were considered. The honest answer includes the ones retrieved and set aside, with the reason.