One-shot prototype
Can the configuration produce a working, constraint-compliant prototype in one attempt?
The PM job
Getting a clickable prototype in front of users or stakeholders fast.
Why it matters
A prototype that looks right but whose core interaction is broken wastes a research session. This task rewards working behaviour over polish.
What good looks like
- The core interaction works end to end
- Respects the stated constraints and scope
- Handles the empty and error states named in the brief
Deliberately not measured
- Production code quality
- Visual taste beyond usability
Turning a spec into a usable interactive artefact
An attractive shell with a broken or absent core interaction
Deterministic checks + blind human review
This task measures the whole configuration. Tools, instructions and skills in the harness contribute to the result, so compare configurations rather than models.
Results
Every evaluated configuration on this task, all cases and repeats.
| # | Model · Harness | Task score | Jev | Martin’s | Runs | Critical failures | Cost / run | Latency |
|---|
Case viewer
Read the brief, then compare up to three outputs side by side.
Build a clickable prototype that lets a patient move an existing appointment to another slot in one flow.
A working rebooking flow with the empty state, within scope.
- Core interaction broken
v1.0 · synthetic · health, mobile