Skip to content

Test a new way of answering customers

Split Conversations between the support seat's current instructions and a new version, with the rule for the winner fixed before it starts.

For developers

Half the Conversations get the Customer Lead's current instructions, and half get the new ones. The Experiment fixes how the winner is picked before the first reply goes out.

For developers

What you get

  • 2versions of the support instructions, each answering half the conversationsVariant
  • 1test with the rule that picks the winner fixed before it startsExperiment
  • 1result after 300 conversations each, naming the winnerDecision

Your agent does

Experiment.create → Experiment.start
You, to your agent

Test a different support prompt against the current one.

Try offering the refund first on half the chats.

Do shorter replies resolve more chats?

# 1. Experiment.create
EXPERIMENT=$(curl -X POST https://api.sb/leaseline.app/experiments \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"variants":[{"name":"control","note":"The Customer Lead’s current contract.","measure":"Conversations resolved with no hand-off that stay resolved for 7 days."},{"name":"refund-first","note":"Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/") | last')

# 2. Experiment.start
curl -X POST https://api.sb/leaseline.app/experiments/$EXPERIMENT/start \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"
import { $ } from 'startup-builder'

const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })
await $.Experiment.start({ experiment: experiment.$id })
EXPERIMENT=$(npx startup-builder do Experiment.create --json '{"variants":[{"name":"control","note":"The Customer Lead’s current contract.","measure":"Conversations resolved with no hand-off that stay resolved for 7 days."},{"name":"refund-first","note":"Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/") | last')
npx startup-builder do Experiment.start --json '{"experiment":"'"$EXPERIMENT"'"}'
{
  "name": "do",
  "arguments": {
    "code": "const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })\nreturn await $.Experiment.start({ experiment: experiment.$id })"
  }
}

Change it

  • Stop the test if refunds double in either arm.
  • Roll the winner out to the support seat.
  • Test the reply length only.

Under the hood

Systems
Experiments, Support, Evals
Nouns
Experiment, Variant, Conversation, Decision
Events
Experiment.started, Experiment.concluded

Wait for the volume. Below a few hundred Conversations per arm a test reads noise, so this wakes once the Startup has them. Until then, read ten transcripts a week and edit the contract by hand.