# Run an experiment

URL: /code/run-an-experiment · Id: `page:code/run-an-experiment`

> Test a challenger prompt against the one in use with two files, read the result per arm, and ship the winner as a Change.

An [Experiment](/business/nouns/experiment) in code is two files: the challenger, beside the value it replaces, and one file in `experiments/` that says what decides it. Merging starts it; concluding it ships the winner.

## Prerequisites
- A project with `npx startup-builder dev` running. The examples use `leaseline.app`, which sells lease-renewal tracking to property managers at $29 a month.
- A prompt in use, `prompts/support-reply.md`, graded by an [Evaluator](/business/nouns/evaluator). [Prompts](/code/file-conventions/prompts) covers the file.

## Steps
1. Copy the prompt to `prompts/support-reply@lease.md` and rewrite its body. The `@lease` suffix makes it an arm, inert until an Experiment names it.

2. Create `experiments/reply-with-the-lease.mdx`:

   ```mdx title="experiments/reply-with-the-lease.mdx"
   ---
   hypothesis: Replies that cite the manager's lease record resolve more Conversations with no person involved.
   varies: prompts/support-reply
   unit: conversation
   metric: agent-resolution-rate
   guardrails: [resolution-time]
   window: 21d
   minN: 120
   ---
   ```

3. Save. The preview shows the Experiment in `draft` and the challenger's Evaluator score beside the control's. A challenger that scores worse fails the [Change](/business/nouns/change) here.

4. Check an assignment without counting it. [Mode](/business/nouns/mode) `try` returns the arm and records no [Exposure](/business/nouns/exposure):

   ```ts
   import { $ } from 'startup-builder'

   const { variant } = await $.try.Flag.evaluate({ flag: 'reply-with-the-lease', unit: 'conversation_1' })
   ```

5. Merge the Change. The challenger runs in shadow on live [Conversations](/business/nouns/conversation), then `Experiment.start` splits them between the arms.

6. Read the result. Each arm's [Readout](/business/nouns/readout) prints its value, its sample and whether it can be read yet:

   ```bash
   npx startup-builder show /experiments/experiment_1
   ```

7. Once the window closes and both arms pass `minN`, conclude it:

**Ask**

```text
Ship the support reply that cites the lease.
```

**API**

```bash
curl -X POST https://api.sb/leaseline.app/experiments/experiment_1/conclude \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"decision":"ship"}'
```

**SDK**

```ts
import { $ } from 'startup-builder'

const { decision } = await $.Experiment.conclude({ experiment: 'experiment_1', decision: 'ship' })
```

**CLI**

```bash
npx startup-builder do Experiment.conclude --json '{"experiment":"experiment_1","decision":"ship"}'
```

**MCP**

```json
{
  "name": "do",
  "arguments": {
    "code": "return await $.Experiment.conclude({ experiment: 'experiment_1', decision: 'ship' })"
  }
}
```

## Result
`ship` records a [Decision](/business/nouns/decision) and opens a Change that makes the challenger `prompts/support-reply.md`. The losing arm is killed and kept, with every Exposure, and the [Hypothesis](/business/nouns/hypothesis) moves to `supported`. `kill` keeps the control; `iterate` drafts the next challenger from the Readout.

A guardrail breach never waits for you: the Experiment stops and every unit returns to the control.

## Next steps
- [Experiments](/code/file-conventions/experiments): every key, and what may never be varied.
- [React hooks](/code/sdk/react): `useVariant` shows an arm in a component.
- [Experimentation](/code/builders/startup/experimentation): how stakes pick the method.