Usability Feedback Synthesizer

Usability Feedback Synthesizer turns raw session transcripts into a first-pass, prioritized list of UX issues with supporting evidence. A researcher (or an orchestration layer) feeds transcripts from moderated or unmoderated studies into an LLM, optionally via tools such as Maze AI or UserTesting AI, and receives structured findings: issue titles, severity or priority suggestions, affected tasks or screens, and short evidence excerpts tied to specific sessions or participants.

The goal is synthesis speed, not automatic truth. The model drafts the issue list so researchers spend less time clustering notes and more time validating patterns, resolving contradictions, and deciding what ships in the research report. Publication still belongs to research: severity, root cause, and recommendations stay under human judgment.

This use case sits in the design stage of software delivery and targets quality outcomes. It pairs naturally with earlier critique and accessibility checks. Related patterns include Accessibility Auto-Audit, API Contract Generator, and Heuristic Design Critique.

Synthesis pays off when you already have transcripts and a clear study goal. Typical fits:

• Multi-session studies where the same tasks recur and issues need clustering across participants
• Mixed moderated and unmoderated programs where note quality varies and a common issue schema helps comparison
• Fast turnarounds between research and design critique, when stakeholders need a provisional issue map before the full write-up

It does not replace observation, affinity mapping when the research question is exploratory, or statistical claims about prevalence. An LLM can over-weight vivid quotes, miss silent struggle that never appears in speech, and invent themes if prompts ask for “top issues” without enough source text. Treat the output as a draft reading of the corpus, not a substitute for watching sessions or reviewing recordings when behavior matters more than what people said.

Hard stop on empty input: if transcripts are missing, incomplete, or only placeholders (empty files, failed exports, consent-redacted blanks with no remaining text), the synthesizer should return empty output. Do not invent issues from task lists, prototypes, or researcher hypotheses alone. No transcripts means no synthesized issue list.

Inputs

• Session transcripts (speaker-labeled when available), with stable session or participant IDs
• Study metadata: tasks, success criteria, prototype or build version, device context
• Optional: researcher notes, severity rubric, known product areas to tag against
• Optional: prior issue taxonomy so naming stays consistent across studies

Outputs

• Prioritized UX issue list (title, short description, suggested priority or severity band)
• Evidence: quotes or paraphrased moments with session IDs and, when possible, task or timestamp references
• Cross-session frequency or clustering hints (for example, “seen in 4 of 8 sessions”), labeled as draft counts from the transcript set only
• Open questions and contradictions the model could not resolve
• Explicit empty result when the transcript set is empty or unusable

Tooling

Platforms such as Maze AI or UserTesting AI can supply transcripts, highlight reels, and AI-assisted summaries. The synthesizer layer should still enforce a fixed schema: issue ID, title, evidence array, priority suggestion, confidence note, and status: empty when there is nothing to analyze. Homegrown pipelines that call a general LLM need the same contract so design and research tools can consume results without free-form prose dumps.

1. Gate on transcripts. Confirm at least one non-empty transcript is present. If not, emit empty output and stop.
2. Normalize the corpus. Strip PII according to your policy, keep speaker labels and session IDs, and attach task markers so evidence can point to the right moment.
3. Extract candidate friction. Ask the model for observable problems (confusion, errors, workarounds, abandoned tasks, conflicting mental models), not generic opinions about the brand.
4. Cluster and name. Merge near-duplicate candidates into issue titles that a design team can act on. Prefer specific labels (“Checkout: discount field clears cart on blur”) over vague ones (“Checkout is confusing”).
5. Attach evidence. Every retained issue needs at least one transcript-backed excerpt or clearly attributed paraphrase with a session ID. Drop issues that cannot be evidenced.
6. Suggest priority, do not finalize it. Use your team’s rubric (impact on task success, frequency across sessions, severity of consequence) as guidance for the model. Mark priority as suggested.
7. Surface uncertainty. List conflicts (participants succeed with different strategies), thin evidence, and tasks with too few observations.
8. Hand off to research. A researcher reviews, merges, re-severitizes, and publishes. The model’s draft is an input to that process, not the published finding set.

Prompting should require evidence-first output and forbid filling gaps with “typical UX problems.” When the corpus is thin, prefer fewer issues with strong evidence over a long speculative list.

Researchers remain accountable for findings. Before an issue list enters a readout, ticket backlog, or design critique:

• Spot-check evidence against the source transcript (and recording when quotes feel off)
• Correct severity using product risk and business context the model cannot see
• Remove false positives driven by one outlier rant or by interviewer leading
• Separate usability issues from feature requests and from bugs that belong in engineering triage
• Decide what is in-scope for this study versus parked for a follow-up

Confidence language matters in stakeholder rooms. Prefer “draft synthesis from N transcripts; research validation pending” over “AI found these critical issues.” Design partners can use the draft to prepare questions for the readout, but commitments and roadmap language should wait until research signs off.

• Empty when empty. Missing transcripts → empty output. Partial batches → synthesize only what exists and label coverage (for example, 5 of 12 sessions transcribed).
• No fabricated stats. Do not invent completion rates, SUS scores, or market benchmarks. Counts come only from the provided sessions.
• Preserve participant dignity. Prefer anonymized IDs; avoid publishing raw quotes that re-identify people in small orgs.
• Keep schema stable. Downstream tools and templates break when issue fields change week to week.
• Log model and prompt versions so later audits can explain why two runs differed.
• Re-run after major transcript corrections (bad diarization fixed, missing session added) rather than silently patching the old list by hand without a trail.

Done well, Usability Feedback Synthesizer shortens the path from session corpus to a reviewable issue map. It does not close the loop: research still decides what is real, how severe it is, and what the team should change next.