Experimentation
Experimentation tests one change against a control, with the threshold fixed before the start.
Outcome
| Part | This Process |
|---|---|
| Outcome | Each Experiment ends at its fixed sample in one recorded Decision: ship, iterate or drop. |
| Reads | Hypothesis |
| Leaves | Experiment, Flag, Decision |
| Runs in | Each Startup |
Accountable
Marketing Lead, Marketing Team.
Workflows
| Workflow | Runs | Team |
|---|---|---|
| Run an experiment | on Experiment.created | Marketing Team |
Who does each step
| Workflow | Step | Role | Calls | Approval |
|---|---|---|---|---|
| Run an experiment | 1. State the Hypothesis as a Metric, a baseline, and a threshold. | Analyst | Hypothesis.create | None |
| Run an experiment | 2. Fix the sample size before the start. | Analyst | Experiment.create | None |
| Run an experiment | 3. Split traffic between a control arm and a test arm with a Flag. | Analyst | Flag.create | None |
| Run an experiment | 4. Read the result only at the fixed sample. | Analyst | Readout.create | None |
| Run an experiment | 5. Record the Decision: ship, iterate, or drop. | Analyst | Experiment.conclude | None |
Events
| Direction | When | What happens |
|---|---|---|
| In | On Experiment.created | Starts Run an experiment |
| Out | Hypothesis.created | Raised by step 1 of Run an experiment |
| Out | Experiment.created | Raised by step 2 of Run an experiment |
| Out | Flag.created | Raised by step 3 of Run an experiment |
| Out | Readout.created | Raised by step 4 of Run an experiment |
| Out | Experiment.concluded | Raised by step 5 of Run an experiment |
Plays
- Smoke test
- Paid search
- Paid social
- Retargeting
- Sponsorships
- Video ladder
- Promote winners
- Reserve drop
- Creator probe
- Creator ads
Controls
No Control names this Process or a Workflow of it.
Settings
| Setting | Type | Default | Say | Set with |
|---|---|---|---|---|
experiments.minSample | integer | 400 | Read no test before 400 visitors per arm | Setting.update |
Failure
A failed step retries; when retries run out, the Action goes to the Marketing Lead.
Loop
| Step | Value |
|---|---|
| Hypothesis | A challenger drafted from the last Readout beats its control more often than a challenger drafted fresh. |
| Measure | Weekly verdicts over 28d, at least 8 observations |
| Learn | The learn step reads the Readout, records Evidence and a Decision, and drafts the next Variant. |
Code
Turn on experimentationEnable
On by default for every Product Template. Process.pause stops new Actions; Actions in progress finish.
Read no test before 400 visitors per armExample
A lesson-booking app tests a new signup page against the current one. It fixes 2,000 Visitors per arm and a 10% lift threshold before the start. At the fixed sample the new page converts 14% better, so the Decision records ship.