Extract discovery insights
Can the model separate evidence, themes and hypotheses without inventing consensus?
The PM job
Turning a stack of call transcripts into what we actually learned.
Why it matters
Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.
What good looks like
- Quotes evidence for each theme and counts sources honestly
- Keeps important dissent visible
- Labels hypotheses as hypotheses
- Says what the research cannot tell us
Deliberately not measured
- Transcript clean-up
- Persona illustration
Faithful synthesis of qualitative research
Invents customer consensus or loses important dissent
Jev + blind human review
Results
Every evaluated configuration on this task, all cases and repeats.
| # | Model · Harness | Task score | Jev | Martin’s | Runs | Critical failures | Cost / run | Latency |
|---|
Case viewer
Read the brief, then compare up to three outputs side by side.
Synthesise these eight discovery calls with finance leads about month-end close. What did we learn?
Themes with honest counts, the two dissenters and the switching regret kept visible, hypotheses labelled.
- Invents a quote
v1.0 · anonymised real · B2B, finance