Skip to content

Test ladder

Every way to grade a Hypothesis, cheapest first, and the rule that sends each one to the lowest rung that can settle it.

The test ladder lists every way to grade a Hypothesis, cheapest first. Each Hypothesis goes to the lowest rung that can yield the grade its threshold needs.

The rungs

Each rung has a typical cost, a time to a Readout, and the strongest Evidence grade it can yield.

RungCostLatencyStrongest gradeReaches peopleWhat it does
Prior lookup$01hOpinionNoPrior lookup reads what the Portfolio already learned about the same Product Template, Motion and state.
Offline replay$04hOpinionNoOffline replay scores a Variant on the kept log of past choices before anyone sees it.
Interview$03dCommitmentYesInterview asks ICP members about the last time the Problem hit, and counts the introductions and time they offer.
Smoke page$01wBehaviorYesSmoke page publishes an Offer at its real Price before it delivers and counts who tries to accept.
Fake door$01wBehaviorYesFake door adds a coming-soon Offer to a live Site and counts the clicks on it.
Pre-sale$02wCommitmentYesPre-sale takes a refundable deposit on a reserve page, priced against what the ICP hires today.
Concierge$1002wBehaviorYesConcierge delivers the outcome by hand to a few paying Customers, in the open.
Wizard of Oz$1002wBehaviorYesWizard of Oz delivers behind the product’s real interface while people do the work.
Paid probe$1001wBehaviorYesPaid probe spends $100 on one ad venue to measure what a click and a Customer cost.
Single capability$1004wBehaviorYesSingle capability ships the one capability the approach rests on and measures use.
Flag split$02wBehaviorYesFlag split randomizes Variants of the built thing behind a Flag and reads each against the control.

The cheapest decisive rung

A rung is decisive when its strongest grade meets the threshold. Among decisive rungs, the Hypothesis takes the one with the lowest cost per day to a verdict.

The two rungs that reach no one run first, on every Hypothesis. They grade Opinion, so they prune weak candidates and never pass a gate. A Prior lookup that finds the question settled across the Portfolio opens the Hypothesis at Testing, not Open.

Where each slot starts

SlotGrade it needsCheapest decisive rung
customerCommitmentInterview: introductions and time offered
problemCommitmentInterview: the last time the Problem hit
approachBehaviorSmoke page at the real Price
switchCommitmentPre-sale: a reserve-first deposit
differentiationBehaviorFake door, one per axis

Before Live, the ladder settles most questions in days. A Flag split waits for a built thing and the traffic to read it.

Lines fixed before the first exposure

Every test states its pass and kill lines before anyone sees it: the Metric, the window and the minimum sample. A line can tighten after launch and never loosen. Test demand runs the rung and records the verdict.

Next

Experimentation explains why every run is a test, and what no test varies.