Tasks / Define

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Measures the modelTask v1.0 · Release · 2 casesDifficulty

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

Makes one clear choice and names what is sacrificed
Grounds the choice in the supplied evidence
Surfaces the tension or contradiction in the data
Defines how we would know it is working

Deliberately not measured

Market sizing accuracy beyond the supplied data
Financial modelling

Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without choices

Grading

Jev + blind human review

Results

Every evaluated configuration on this task, all cases and repeats.

#	Model · Harness	Task score	Jev	Martin’s	Runs	Critical failures	Cost / run	Latency

Case viewer

Read the brief, then compare up to three outputs side by side.

The brief

Our tutoring marketplace has stalled at 4,000 monthly bookings. Write a one-page strategy for the next 12 months. We can fund one major bet.

MetricsTutor supply up 60% YoY; bookings up 8%. 70% of tutors get zero bookings per month. Repeat booking rate 64% for students who book twice.

ResearchStudents cite 'finding the right tutor' as the top problem. Tutors cite 'not enough students'.

Board viewThe board believes paid acquisition of students is the answer.

What a strong answer does

Recognise the contradiction: supply is abundant but matching fails, so buying demand into a broken match pours water into a leaky bucket. Choose matching quality and say what is given up.

Critical failures (cap the score)

Recommends paid demand acquisition without addressing matching

Case

v1.0 · synthetic · marketplace, board pressure