Opportunity Scoring Model

An opportunity scoring model ranks items in a software discovery backlog by predicted impact on outcomes you already measure, typically revenue lift, retention, activation, or related quality signals. It learns from historical pairs of delivered work and the outcomes that followed, then scores new candidates so the highest-ranked items rise for review.

The model does not ship features or lock a roadmap. It produces an ordered list with scores and, when available, feature attributions that explain why an item ranked high or low. A product manager still decides what enters a commitment cycle, what stays in research, and what gets deferred.

Treat the score as a prioritization aid under uncertainty, not as a forecast you can put in a board deck without caveats. Prediction quality tracks how cleanly past delivery maps to measured outcomes in your product. Teams with sparse, noisy, or delayed outcome labels get weaker rankings and should weight judgment more heavily.

For adjacent discovery signals that feed or compete with scored opportunities, see Competitive Intelligence Monitor and Feature Request Clustering.

The system should return empty ranked output when usable outcome history is missing. Empty means no scores, no ordered backlog slice, and no implied priority. A partial table of ideas without labels is not enough to invent impact.

Outcome history is missing when you lack a reliable join between past backlog items (or shipped changes) and post-release metrics over a window long enough for the outcome to appear. Common gaps include no instrumentation for retention or revenue attribution, delivery records that cannot be mapped to experiments or releases, and outcomes that only exist as anecdotes.

Empty output protects the PM from false precision. Ranking without labels encourages narrative scoring dressed up as ML. Prefer an explicit empty state and a checklist of what history to collect over a default sort by votes, recency, or stakeholder volume.

Once labels exist for a minimum cohort of completed items with comparable outcome windows, scoring can run. Until then, keep discovery qualitative and document assumptions with tools such as a Stakeholder Assumption Extractor rather than fabricating predicted impact.

Useful rankings depend on three linked data layers: candidate features, historical delivery context, and outcome labels.

Candidate features describe each backlog item as it sits in discovery. Typical fields include problem statement, segment or persona, estimated effort band, request volume or cluster size, strategic theme, dependency flags, and links to research. Clusters from Feature Request Clustering often become the unit scored, so you rank coherent opportunities instead of duplicate tickets.

Historical delivery context records what shipped before: release dates, scope proxies, experiment flags, and which segments saw the change. Without this join key, the model cannot learn which kinds of work correlated with which outcomes.

Outcome labels are the supervised target. Choose one primary outcome per scoring run when possible, for example retention at a fixed horizon or revenue attributable to a cohort. Secondary outcomes can appear as separate score columns so the PM sees tradeoffs. Labels must share a consistent definition and time window across the training set.

Quality of input beats volume of input. A smaller set of well-joined shipped items with trusted metrics outperforms a large backlog of unlabeled ideas. Exclude cancelled or never-shipped items from training unless you have a deliberate negative-label policy. Exclude vanity metrics that do not reflect the quality outcome you care about.

Run scoring on a stable snapshot of the discovery backlog, then review the ranked list in a commitment ritual, not in an automated pipeline that writes to the roadmap.

Start with the top band and the bottom band. Confirm that high scores match opportunities with plausible causal paths to the labeled outcome. Spot-check low scores that stakeholders treat as must-haves; those conflicts are often where strategy, compliance, or competitive pressure sits outside the training distribution. Competitive pressure itself may surface from a Competitive Intelligence Monitor and should be weighed explicitly rather than hoped into the model.

Use explanations when the system provides them. Feature importance or similar attributions help you see whether the model is overweighting request volume, segment size, or effort. If explanations keep pointing at a proxy you do not trust, pause scoring and fix labels or features before ranking again.

Commit only after human review. Acceptance criteria for a commit might include a clear problem statement, a measurable success metric aligned with the scored outcome, and an owner for post-release readout. The score informs sequencing; the PM owns the bet.

After delivery, close the loop. Record the actual outcome against the item so the next training set improves. Items that ranked high but failed to move the metric are as valuable as successes for recalibration.

Scoring inherits bias from what you have shipped. If past roadmaps favored one segment, predictions will favor similar work even when another segment holds more upside. Correct with explicit segment-aware labels or with PM overrides that you log for audit.

Lagging outcomes create timing errors. Retention and revenue often move weeks after release. If training windows are too short, the model learns noise. If they are too long, confounding changes pile up. Document the window and keep it stable across runs.

Proxy collapse is common. Request volume, NPS comments, and sales anecdotes correlate imperfectly with retention or revenue. When those proxies dominate features, rankings look confident and still miss quality outcomes. Prefer outcome labels over proxy targets whenever instrumentation allows.

Distribution shift breaks transfer. A new platform, pricing change, or market entry can make historical relationships stale. Detect shift by monitoring prediction calibration on recent deliveries; when calibration drifts, retrain or suspend automated ranking until you have post-change labels.

Never treat empty history as a soft warn with a fallback sort. Empty means stop. Fallback sorts reintroduce the politics and volume bias the model was meant to reduce.

Opportunity scoring sits downstream of structured discovery inputs and upstream of commitment. Cluster raw requests before scoring so each row is a coherent opportunity (Feature Request Clustering). Surface external moves that may justify overrides when the model underweights strategic threats (Competitive Intelligence Monitor). Capture unspoken beliefs that drive manual reordering so you can test them later (Stakeholder Assumption Extractor).

Keep the operating rule simple: the model ranks; the PM commits; missing outcome history yields empty output. That discipline keeps discovery backlog scoring useful for quality-oriented prioritization without pretending prediction replaces product judgment.