Domain-specific discovery

Analysts reach for semantic search when keyword queries keep returning either forty thousand hits or none. The useful output is a list of candidate studies, each with an identifier, a title, a venue, a date, and the query that surfaced it. That list goes into screening. It is not a finding, and the paragraph the tool writes above the list is not a finding either.

Tools in this class sit on different indexes and answer different questions. Elicit and Consensus search published research and return per-paper extractions. Exa runs neural search over the open web, which reaches material that never entered a bibliographic database. Research Rabbit works the citation graph, expanding outward from papers already known to be relevant. None of them holds the same documents. The difference between them is coverage, not quality.

That is why this step pays for itself cheaply. The corpus choice does most of the work, and choosing it costs an hour of thought rather than a week of engineering.

What these tools do not do is decide whether a study answers the question. They rank by similarity to the phrasing. A weak paper written in the exact vocabulary of the query outranks a strong one that used different words. The question itself and the criteria that screen against it come from upstream, from a brief converted into a hypothesis tree and screening criteria.

Write down which body of work would contain a true answer, then pick the index that holds it. A question about measured effects lives in the trial and journal literature. A question about how a program was actually delivered lives in evaluation reports and agency documents that bibliographic databases never indexed. A question about who is working on something right now lives in the citation graph and on preprint servers.

Four settings decide most of the recall. Date window, with a stated reason: a policy change, a technology shift, the year an outcome measure was standardized. Language and setting, because a restriction to one language is a real exclusion criterion and belongs in the write-up. Design constraints, if the question only accepts certain study types. Seed papers, for graph expansion, chosen by name.

Phrase the query as a claim, not as keywords. Semantic retrieval works on the meaning of a sentence. "Small-group reading instruction delivered by trained volunteers improves decoding in early primary grades" retrieves differently from "tutoring literacy outcomes," and better.

When nobody can say which corpus would hold the answer, stop and go back to the question. Running a broad search over a general index produces a long list that feels like progress and cannot be defended in a methods section.

Every row needs an identifier a second person can resolve: a DOI, a registry number, a stable repository link. An author and a year is not enough, because near-identical titles are common and reports get reissued.

Research search tools extract per-paper fields, often sample size, population, design, and outcome. Those extractions speed up screening and they are claims about the paper, not the paper. Store each one next to the sentence it was drawn from, or store it marked unverified. When the extraction and the abstract disagree, the abstract wins and the row goes to a human. The same discipline runs later at scale in checking whether a cited passage supports the claim.

Do not let the tool's answer paragraph travel with the list. It reads like a conclusion, it was assembled from whichever hits ranked highest, and it says nothing about what the index did not contain.

Record the run itself: tool, exact query string, filters, date, and the number of results before and after filtering. Semantic search is not reproducible across months. Indexes grow, ranking models change, and a query rerun in November returns a different set than it did in June. An undated run cannot be repeated or challenged.

The walkthrough is hypothetical and the numbers are illustrative. A foundation's evidence analyst is screening for a funding round on tutoring for children in their first three years of school. The locked question: which tutoring designs have measured reading outcomes, and what separates the ones with effects from the ones without.

The first corpus is published research, queried as a claim sentence with a date window opening at the year the reading measure in scope became standard. It returns 240 candidates with per-paper extraction on grade band, dosage, tutor type, and outcome measure. The screening rules, run as a rules pass that rejects candidates failing hard criteria, require a measured reading outcome and a comparison condition. Seventy one survive.

The second corpus is the citation graph, seeded with six papers the program team already trusts. It returns 130 records, 40 of them already in the first set. Of the remaining 90, eleven survive the same rules. Nine of those eleven were missed the first time because they called the intervention small-group reading instruction and never used the word tutoring anywhere in the abstract.

The third corpus is neural web search, aimed at delivery rather than effects. It returns fourteen items: district evaluation reports, a state audit, and two program manuals. None appears in either other set, and none would ever enter a bibliographic index. These do not join the effects list. They open a separate line for the sweep that handles material like this.

The analyst reads 96 abstracts. The nine vocabulary misses are the argument for running the graph, and the fourteen web items are the only documents describing what tutors did, for how many minutes, with what training.

The summary paragraph is the first trap. It is a readable digest of whatever ranked highest, and its confidence is unrelated to whether the index held the relevant work. A confidently incomplete answer is harder to catch than an empty result, because nothing about it looks wrong.

Vocabulary lock-in survives the move to semantic search. Retrieval on meaning helps, but subfields name the same intervention differently, and a query written in one field's language still under-reaches another's. Run the claim in at least two vocabularies and compare what each one adds.

Citation-graph expansion reproduces a school of thought. Seeds chosen from work the team already believes will return more of the same, including the shared assumptions and the shared blind spots. Seed with at least one paper from a group that reached a different conclusion.

Silent filters are the quiet one. Some tools apply their own inclusion rules, restricting to certain venue types or excluding preprints by default. An undocumented filter is an exclusion criterion nobody wrote down. Read the settings and copy them into the search log.

Material that never entered a database needs its own pass. Reports, registry entries, and corrections belong in a targeted sweep of practitioner and registry sources, which is read against different standards.

A retrieved row becomes a screening candidate when a named person accepts it, not when the tool returns it. Retrieval speed makes it tempting to move the whole export forward. A list nobody triaged is a backlog with a citation count attached.

Publish the search log with the candidate list. Corpus, query string, filters, date, counts before and after screening, and the corpora that were considered and skipped. The reasons a corpus was skipped matter as much as the results, because that is where a reviewer will find the gap.

Say what was not searched. If subscription access blocked an index, if a language was excluded, if the graph was seeded from one group's work, write it down as a limitation rather than leaving the coverage implied. A silent gap becomes a claim about the field.

Where breadth matters more than corpus fit, this runs alongside the same question put through several deep-research engines, which trades precision for reach. The two produce different lists on purpose. Merge them at screening, under one person's name, with the origin of every row still visible.