Experimentation
Every change a Startup makes runs as an Experiment, and the winning variant becomes the definition that later Startups inherit.
A Startup runs each change as an Experiment: a Hypothesis goes in, variants run against a Metric, and a Decision comes out. The winner becomes the definition, and the Studio runs the same loop across its Portfolio.
Every run is a loop
Experimentation is the resting state of a Startup, not a program someone starts. Every Play run carries a loop, and so does every Process run that reaches a Customer or a prospective one.
The loop names a Hypothesis, a Metric with a window and a minimum sample, and a learn step that no definition can remove. The learn step records Evidence and a Decision: ship, kill, or iterate. On iterate, it drafts the next variant as a proposed Change.
A Process with nothing to learn, such as a month-end close, declares no loop and cites the control it serves. Everything else learns by default, so a Startup that sends, publishes, or sells is always testing something.
What varies
Any value a Startup owns can hold variants in place of one answer. A Flag assigns each unit its variant by a keyed hash, so a Visitor sees the same headline on every visit.
| What varies | Example | Unit | Pinned on |
|---|---|---|---|
| Offer terms | a 14-day trial with no card | Visitor | the Order |
| Price | $29 against $39, as sibling Offers | new Customer | the Order |
| Page copy | a headline that names the Problem | Visitor | the Contact at signup |
| Message | a cold email subject line | Contact | the Sequence |
| Play mix | spend split between two Plays | Startup | the week's budget |
| Agent behavior | a Role agent's instructions or Skill | Conversation | the Action |
| Interview script | the opening question | Contact | the Interview |
An Exposure is recorded when a variant is consumed: rendered, sent, charged, or run. A Metric counts only from first exposure.
Agent behavior clears two gates before any Customer sees it. A candidate first passes its Evaluators offline at no worse than the current version, then runs in shadow on live inputs with no one-way Actions. Only then does it run online.
Stakes choose the method
The stakes of the value being varied decide how it is tested and who ships the winner.
| Stakes | Examples | How it runs | Ship waits |
|---|---|---|---|
| Low | copy, send time, onboarding order | a bandit that shifts traffic to the leader | nothing: any agent ships |
| Moderate | trial length, signup gate, agent instructions | a pre-registered sequential test | one business hour |
| High | Price, refund Limits, Offer terms | a fixed or sequential test, never a bandit | one business day, or an Approval below full autonomy |
| Never | legal floor, Consent, AI disclosure, refund rights, identity | none: the Change is refused | not applicable |
Delay windows hold the decision, not the exposure, and watchers can pause or cancel inside them. Guardrail Metrics stop an Experiment the moment one breaches, and the control returns to every unit. Guardrail stops never wait, and anyone with access can stop any Experiment.
An agent never varies its own yardstick. The Metric definitions, Evaluators, and guardrails that judge an Experiment are fixed for the agent that runs it.
Price gets the strictest rule. Two Customers in one market never see different prices for the same Offer at the same moment. A price test runs as a sibling Offer for new Customers, by region, or in alternating weeks.
Small traffic
A new Startup with 40 Visitors a week cannot settle a split test in any useful time. Three moves give it a verdict anyway:
- Ask the Portfolio first. Sibling Startups with the same Product Template, Motion, and Lifecycle state have already learned things, and their results open every new test.
- Pool across siblings. Sibling Startups testing the same value read as strata of one Experiment.
- Take the cheapest decisive test. A deposit page settles in two weeks what a split test settles in a quarter. Test demand lists the rungs.
Every number prints its readability, such as "Visitor to deposit: 37 of 200 exposures, readable in 9 days." A Startup never ships a high-stakes winner on a number that cannot be read.
Winners become the definition
Shipping writes one Change where the Experiment sat, replacing the Flag with the winning value. A Startup's own Experiment changes its own definition. A Studio Experiment on a default writes a new Version of the Startup Template.
Losing variants are killed and kept: their exposures and Evidence stay on the record forever. The Hypothesis moves to Supported or Refuted, which stops the next agent from testing it again.
The Decision records its premise: the measured effect and the conditions it held under. When traffic grows, the buyer mix shifts, or a year passes, the premise lapses. The shipped value then becomes the control of a new Experiment.
A sales-tax API for developers tests two docs pages: one opens with a request example, the other with a pricing table. The request example lifts signups from 3.1% to 4.4% over 2,100 Visitors. It ships, and its Decision notes that it held for b2d traffic from search.
The Studio runs the same loop
One level up, the variants are whole Startups, Startup Template defaults, and Play Versions.
An Idea spawns three to five sibling Startups as one Experiment. Each varies the riskiest part of its Hypothesis, such as the ICP, the approach, or the price band. Each opens a reserve page on day one.
Deposits decide. The winner is promoted, and the others go Dormant, kept and revivable.
A Play rolls out by Cohort: 1%, then 10%, 50%, and all, with a control Cohort kept on the prior Version for the whole run. A guardrail breach reverts every write the Play made, in every Startup, in one query.
Winning variants flow back into Startup Template defaults, so every new Startup starts from the best known version. Each default records the Experiment that set it and becomes the control of the next one. Speed is local and rigor is global: a write to a shared default first passes a Portfolio-wide false-discovery check.
The Studio also keeps a program holdout: a small share of Startups stays on the root defaults for a quarter. Their results measure whether experimenting pays.
Next
- Test demand: the cheapest test that settles each Hypothesis.
- Wants: unmet demand as Evidence, and deposits before building.
- Office hours: where the week's Decision is written.
- Experiments: the System that runs every Experiment.
- Plays, cohorts, and experiments: the Studio's side of the loop.