Experiments
Experiments tests Hypotheses by varying copy, Price or behavior through Flags and decides each winner from primary and guardrail Metrics.
Experiments tests a Hypothesis by splitting traffic across variants of copy, Offers or behavior through Flags. Primary and guardrail Metrics pick the winner, and the winner becomes the definition.
Summary
| Part | This System |
|---|---|
| Group | Data and insights |
| Status | Beta: it works, and its Verbs may still change |
| Shelf | Front: offered from a Startup’s first days |
You can ask:
- “Test a $39 price against $29”
- “Try a new headline on half the visitors”
- “Which version won?”
- “Stop the test”
What you get
| Part | This System |
|---|---|
| Starts | On for every Startup at creation, with nothing to install. |
| Built from | Event streams, Functions, Database, Analytics |
Nouns it owns
| Noun | Definition |
|---|---|
| Experiment | An Experiment is a test of a Hypothesis that varies copy or behavior through Flags and judges the variants by a primary Metric. |
| Hypothesis | A Hypothesis is a belief about a Startup stated so it can be proved wrong, with the Metric and threshold that settle it. |
| Variant | A Variant is one value an Experiment can assign, such as a revision, a Price on a sibling Offer, or a prompt, with its lineage. |
| Exposure | An Exposure is the moment one unit meets the Variant its Flag assigned, logged with every option and its propensity. |
| Readout | A Readout is an Experiment’s result per Variant on its Metric, with the sample size, the interval, and how far the number can be trusted. |
Verbs
Events
| Direction | Event | How |
|---|---|---|
| Emits | Experiment.started | By Experiment.start |
| Emits | Experiment.concluded | By Experiment.conclude |
| Emits | Hypothesis.tested | By Hypothesis.test |
| Emits | Hypothesis.dropped | By Hypothesis.drop |
States
The lifecycle of an Experiment:
| State | Meaning | Moves to |
|---|---|---|
draft (first) | Its variants and Metrics are set, and no traffic is split yet. | running by Experiment.start |
running | Traffic is split across its variants while its Metrics are measured. | concluded by Experiment.conclude |
concluded | It ended with a Decision to ship, kill, or iterate. | None: it ends here. |
Its other Nouns with a lifecycle: Hypothesis.
Who operates it
| Part | This System |
|---|---|
| Accountable | Analyst |
| Team | Marketing Team |
| Seat | A role agent holds it until a Human is seated. |
| Autonomy | supervised |
| Also calls its Verbs | Strategist |
Approvals and limits
None.
Processes that use it
| Process | How it uses it |
|---|---|
| Pricing | A step calls Experiment.start |
| Experimentation | A step calls Experiment.conclude |
| Product definition | A step calls Experiment.start |
Rail underneath
None.
Views and dashboards
| View | Shows | Metrics |
|---|---|---|
| The Analyst’s view | The Metrics its accountable Role answers for. | Metric freshness |
Access
What each Access level may call here.
| Access level | May call |
|---|---|
owner | Every Verb (23) |
admin | Every Verb (23) |
member | Every Verb (23) |
viewer | Read only (10 of 23) |
Import and export
None.
Fees and pricing
None.
Availability and limits
| Part | This System |
|---|---|
| Status | Beta: it works, and its Verbs may still change |
Compliance notes
Controls it carries:
Related Systems
Example
A Directory of AI tools tests two headlines on the Page where vendors buy listings. After 1,900 Visits, the second headline lifts purchases 22% while refunds, the guardrail, hold flat. The Analyst Agent concludes the Experiment, and the losing variant stays on record with its Evidence.