Skip to content
StartupBuilderby Startups.Studio
Code
Changelog
File conventionsAgents and Workflows

Experiments

A file in experiments/ is one Experiment, the Hypothesis, what it varies, who is assigned and the Metrics that judge it, started on merge.

A file in experiments/ is one Experiment: the Hypothesis under test, the value it varies, the unit assigned, and the Metrics that pick the winner. It implements Experimentation.

  • leaseline/
  • experiments/
  • reply-with-the-lease.mdxone Experiment
  • prompts/
  • support-reply.mdthe control
  • support-reply@lease.mdthe challenger arm

Reference

MDX. Frontmatter is the Experiment; the body is the reasoning, which the Decision cites when it concludes.

KeyTypeHolds
hypothesisstringThe belief, stated so the primary Metric can prove it wrong. Required
variespath or keyWhat changes: a file such as prompts/support-reply, an Offer, a Page, or a Setting such as settings.trial. Required
armsmapEach arm's value; control is the current one. A file's arms are its @arm siblings instead
unitvisitor | customer | contact | conversation | requestWho is assigned. Set by what varies when omitted
splitmapEach arm's share of units. Even by default
metricMetric idThe primary Metric. Required
guardrailsMetric idsMetrics that stop the Experiment the moment one breaches
window, minNduration, integerHow long it runs, and the fewest exposed units per arm before a verdict

Examples

Minimal

experiments/trial-length.mdx
---
hypothesis: A 30-day trial turns more managers into paying Customers than a 14-day trial.
varies: settings.trial
arms: { control: 14d, month: 30d }
metric: outside-paying-customers
---

Leaseline

Leaseline's support Agent answers property managers. The challenger reply cites the manager's own lease record before it explains anything.

experiments/reply-with-the-lease.mdx
---
hypothesis: Replies that quote the manager's lease record resolve more Conversations with no person involved than replies that explain the feature.
varies: prompts/support-reply
unit: conversation
metric: agent-resolution-rate
guardrails: [resolution-time]
window: 21d
minN: 120
---
Most questions are about one lease. Showing that lease answers the question the manager asked.

Code that branches on an arm asks the Flag the file compiles to. The answer is the same for a unit every time, and the Exposure is recorded for you:

TypeScript
import { $ } from 'startup-builder'

const { variant } = await $.Flag.evaluate({ flag: 'reply-with-the-lease', unit: 'conversation_1', default: 'control' })

$.choose('reply-with-the-lease', { unit }) is the same call, resolving to the arm's key.

What it builds

On merge, the StartupBuilder compiles the file into a running test.

  • a Hypothesis, bound with Hypothesis.test
  • an Experiment in draft, with one Variant per arm and control marked
  • a Flag that assigns each unit by a keyed hash, read by Flag.evaluate
  • Experiment.start, once each arm clears its gates; an Agent's instructions first pass their Evaluators offline, then run in shadow
  • an Exposure when a unit meets its arm, and a Readout per arm each day
  • Experiment.conclude with ship, kill or iterate; ship opens a Change that makes the winner the file

Good to know

  • The stakes of what varies pick the method. A Price, a refund Limit or Offer terms never run as a bandit, and their winner waits a business day.
  • Two Customers in one market never see two Prices for one Offer. Test a Price as a sibling Offer for new Customers.
  • The legal floor, Consent, AI disclosure and refund rights are never varied: a file that varies them is refused at its Change.
  • The Agent running an Experiment never edits its metric or guardrails.
  • A losing arm is killed and kept, with every Exposure on the record.

Plays

Previous Page

Prompts

Next Page