Run an experiment

An Experiment in code is two files: the challenger, beside the value it replaces, and one file in experiments/ that says what decides it. Merging starts it; concluding it ships the winner.

PrerequisitesCopy Anchor Link

• A project with npx startup-builder dev running. The examples use leaseline.app, which sells lease-renewal tracking to property managers at $29 a month.
• A prompt in use, prompts/support-reply.md, graded by an Evaluator. Prompts covers the file.

StepsCopy Anchor Link

1. Copy the prompt to prompts/support-reply@lease.md and rewrite its body. The @lease suffix makes it an arm, inert until an Experiment names it.

2. Create experiments/reply-with-the-lease.mdx:

   Copy Text

   ---
   hypothesis: Replies that cite the manager's lease record resolve more Conversations with no person involved.
   varies: prompts/support-reply
   unit: conversation
   metric: agent-resolution-rate
   guardrails: [resolution-time]
   window: 21d
   minN: 120
   ---

3. Save. The preview shows the Experiment in draft and the challenger's Evaluator score beside the control's. A challenger that scores worse fails the Change here.

4. Check an assignment without counting it. Mode try returns the arm and records no Exposure:

   Copy Text

   import { $ } from 'startup-builder'
   
   const { variant } = await $.try.Flag.evaluate({ flag: 'reply-with-the-lease', unit: 'conversation_1' })

5. Merge the Change. The challenger runs in shadow on live Conversations, then Experiment.start splits them between the arms.

6. Read the result. Each arm's Readout prints its value, its sample and whether it can be read yet:

   Copy Text

   npx startup-builder show /experiments/experiment_1

7. Once the window closes and both arms pass minN, conclude it:

Ship the support reply that cites the lease.

ResultCopy Anchor Link

ship records a Decision and opens a Change that makes the challenger prompts/support-reply.md. The losing arm is killed and kept, with every Exposure, and the Hypothesis moves to supported. kill keeps the control; iterate drafts the next challenger from the Readout.

A guardrail breach never waits for you: the Experiment stops and every unit returns to the control.

Next stepsCopy Anchor Link

• Experiments: every key, and what may never be varied.
• React hooks: useVariant shows an arm in a component.
• Experimentation: how stakes pick the method.