Test a new way of answering customers
Split Conversations between the support seat's current instructions and a new version, with the rule for the winner fixed before it starts.
For developers
Half the Conversations get the Customer Lead's current instructions, and half get the new ones. The Experiment fixes how the winner is picked before the first reply goes out.
For developers
What you get
- 2versions of the support instructions, each answering half the conversationsVariant
- 1test with the rule that picks the winner fixed before it startsExperiment
- 1result after 300 conversations each, naming the winnerDecision
Your agent does
You, to your agent
Test a different support prompt against the current one.
Try offering the refund first on half the chats.
Do shorter replies resolve more chats?
# 1. Experiment.create
EXPERIMENT= $(curl -X POST https: //api.sb/ leaseline.app/ experiments \
-H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
-H "Content-Type: application/ json" \
-d '{"variants": [{"name": "control", "note": "The Customer Lead’s current contract.", "measure": "Conversations resolved with no hand-off that stay resolved for 7 days."}, {"name": "refund-first", "note": "Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/ ") | last')
# 2. Experiment.start
curl -X POST https: //api.sb/ leaseline.app/ experiments/ $EXPERIMENT/ start \
-H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"
import { $ } from 'startup-builder'
const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })
await $.Experiment.start({ experiment: experiment.$id })
EXPERIMENT= $(npx startup-builder do Experiment.create --json '{"variants": [{"name": "control", "note": "The Customer Lead’s current contract.", "measure": "Conversations resolved with no hand-off that stay resolved for 7 days."}, {"name": "refund-first", "note": "Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/ ") | last')
npx startup-builder do Experiment.start --json '{"experiment": "'"$EXPERIMENT"'"}'
{
"name": "do",
"arguments": {
"code": "const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })\nreturn await $.Experiment.start({ experiment: experiment.$id })"
}
}
Change it
- Stop the test if refunds double in either arm.
- Roll the winner out to the support seat.
- Test the reply length only.
Under the hood
- Systems
- Experiments, Support, Evals
- Nouns
- Experiment, Variant, Conversation, Decision
- Events
Experiment.started,Experiment.concluded
Wait for the volume. Below a few hundred Conversations per arm a test reads noise, so this wakes once the Startup has them. Until then, read ten transcripts a week and edit the contract by hand.