# Experiments

URL: /code/file-conventions/experiments · Id: `page:code/file-conventions/experiments`

> A file in experiments/ is one Experiment, the Hypothesis, what it varies, who is assigned and the Metrics that judge it, started on merge.

A file in `experiments/` is one [Experiment](/business/nouns/experiment): the [Hypothesis](/business/nouns/hypothesis) under test, the value it varies, the unit assigned, and the [Metrics](/business/nouns/metric) that pick the winner. It implements [Experimentation](/code/builders/startup/experimentation).

```text
leaseline/
├── experiments/
│   └── reply-with-the-lease.mdx    one Experiment
└── prompts/
    ├── support-reply.md            the control
    └── support-reply@lease.md      the challenger arm
```

## Reference
MDX. Frontmatter is the Experiment; the body is the reasoning, which the [Decision](/business/nouns/decision) cites when it concludes.

| Key | Type | Holds |
| --- | --- | --- |
| `hypothesis` | string | The belief, stated so the primary Metric can prove it wrong. Required |
| `varies` | path or key | What changes: a file such as `prompts/support-reply`, an [Offer](/business/nouns/offer), a [Page](/business/nouns/page), or a [Setting](/business/nouns/setting) such as `settings.trial`. Required |
| `arms` | map | Each arm's value; `control` is the current one. A file's arms are its `@arm` siblings instead |
| `unit` | `visitor \| customer \| contact \| conversation \| request` | Who is assigned. Set by what varies when omitted |
| `split` | map | Each arm's share of units. Even by default |
| `metric` | [Metric](/business/nouns/metric) id | The primary Metric. Required |
| `guardrails` | Metric ids | Metrics that stop the Experiment the moment one breaches |
| `window`, `minN` | duration, integer | How long it runs, and the fewest exposed units per arm before a verdict |

## Examples
### Minimal
```mdx title="experiments/trial-length.mdx"
---
hypothesis: A 30-day trial turns more managers into paying Customers than a 14-day trial.
varies: settings.trial
arms: { control: 14d, month: 30d }
metric: outside-paying-customers
---
```

### Leaseline
Leaseline's support [Agent](/business/nouns/agent) answers property managers. The challenger reply cites the manager's own lease record before it explains anything.

```mdx title="experiments/reply-with-the-lease.mdx"
---
hypothesis: Replies that quote the manager's lease record resolve more Conversations with no person involved than replies that explain the feature.
varies: prompts/support-reply
unit: conversation
metric: agent-resolution-rate
guardrails: [resolution-time]
window: 21d
minN: 120
---
Most questions are about one lease. Showing that lease answers the question the manager asked.
```

Code that branches on an arm asks the [Flag](/business/nouns/flag) the file compiles to. The answer is the same for a unit every time, and the [Exposure](/business/nouns/exposure) is recorded for you:

```ts
import { $ } from 'startup-builder'

const { variant } = await $.Flag.evaluate({ flag: 'reply-with-the-lease', unit: 'conversation_1', default: 'control' })
```

`$.choose('reply-with-the-lease', { unit })` is the same call, resolving to the arm's key.

## What it builds
On merge, the [StartupBuilder](/code/builders/startup) compiles the file into a running test.

- a Hypothesis, bound with `Hypothesis.test`
- an Experiment in `draft`, with one [Variant](/business/nouns/variant) per arm and `control` marked
- a Flag that assigns each unit by a keyed hash, read by `Flag.evaluate`
- `Experiment.start`, once each arm clears its gates; an Agent's instructions first pass their [Evaluators](/business/nouns/evaluator) offline, then run in shadow
- an Exposure when a unit meets its arm, and a [Readout](/business/nouns/readout) per arm each day
- `Experiment.conclude` with `ship`, `kill` or `iterate`; `ship` opens a [Change](/business/nouns/change) that makes the winner the file

## Good to know
- The stakes of what varies pick the method. A [Price](/business/nouns/price), a refund [Limit](/business/nouns/limit) or Offer terms never run as a bandit, and their winner waits a business day.
- Two [Customers](/business/nouns/customer) in one market never see two Prices for one Offer. Test a Price as a sibling Offer for new Customers.
- The legal floor, [Consent](/business/nouns/consent), AI disclosure and refund rights are never varied: a file that varies them is refused at its Change.
- The Agent running an Experiment never edits its `metric` or `guardrails`.
- A losing arm is killed and kept, with every Exposure on the record.