# Test a new way of answering customers

URL: /agents/test-new-support-instructions · Id: `guide:test-new-support-instructions`

> Split Conversations between the support seat's current instructions and a new version, with the rule for the winner fixed before it starts.

Half the [Conversations](/business/nouns/conversation) get the [Customer Lead](/agents/customer/customer-lead)'s current instructions, and half get the new ones. The [Experiment](/business/nouns/experiment) fixes how the winner is picked before the first reply goes out.

**Before it runs**

- You leave the chat: Never: it all happens in the chat.

## What you get
- 2 versions of the support instructions, each answering half the conversations (Variant)
- 1 test with the rule that picks the winner fixed before it starts (Experiment)
- 1 result after 300 conversations each, naming the winner (Decision)

## Your agent does
**For agents**

- Guide: `guide:test-new-support-instructions`, status `live`. Page: [/agents/test-new-support-instructions.md](/agents/test-new-support-instructions.md).
- Approval: none. Run it when the founder asks.
- Leaves the chat: never.
- Run: call the steps in order, each through the API, SDK, CLI or MCP tab below; a later step passes the `$id` an earlier one returns. Each value marked example is the sample business’s: pass the founder’s own.

| Step | Verb | Inputs (type, required, this guide’s value) | Returns |
| --- | --- | --- | --- |
| 1 | [`Experiment.create`](/business/nouns/experiment/create) (`verb:Experiment.create`) | `variants` json, required: example `[{"name":"control","note":"The Customer Lead’s current contract.","measure":"Conversations resolved with no hand-off that stay resolved for 7 days."},{"name":"refund-first","note":"Offer the refund before asking why, on Orders under $50."}]` | `experiment` Experiment |
| 2 | [`Experiment.start`](/business/nouns/experiment/start) (`verb:Experiment.start`) | `experiment` Experiment (an id), required: the `$id` step 1 returns | `experiment` Experiment |

Returns: the last call answers with `experiment` and its `action`, whose `status` is `done`, `pending` (waiting on an Approval or a Worker), `failed`, `canceled` or `refused`:

```json
{
  "api": "https://api.sb/leaseline.app",
  "experiment": {
    "$id": "https://api.sb/leaseline.app/experiments/experiment_1",
    "$type": "Experiment",
    "variants": [
      {
        "name": "control",
        "note": "The Customer Lead’s current contract.",
        "measure": "Conversations resolved with no hand-off that stay resolved for 7 days."
      },
      {
        "name": "refund-first",
        "note": "Offer the refund before asking why, on Orders under $50."
      }
    ]
  },
  "action": {
    "$id": "https://api.sb/leaseline.app/actions/action_1",
    "operation": "Experiment.start",
    "status": "done"
  },
  "links": {
    "self": "https://api.sb/leaseline.app/actions/action_1"
  }
}
```

Next: offer “Stop the test if refunds double in either arm”, or “Roll the winner out to the support seat”.

Calls: `Experiment.create` → `Experiment.start`

Ask:

```text
Test a different support prompt against the current one.
Try offering the refund first on half the chats.
Do shorter replies resolve more chats?
```

API:

```bash
# 1. Experiment.create
EXPERIMENT=$(curl -X POST https://api.sb/leaseline.app/experiments \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"variants":[{"name":"control","note":"The Customer Lead’s current contract.","measure":"Conversations resolved with no hand-off that stay resolved for 7 days."},{"name":"refund-first","note":"Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/") | last')

# 2. Experiment.start
curl -X POST https://api.sb/leaseline.app/experiments/$EXPERIMENT/start \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY"
```

SDK:

```ts
import { $ } from 'startup-builder'

const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })
await $.Experiment.start({ experiment: experiment.$id })
```

CLI:

```bash
EXPERIMENT=$(npx startup-builder do Experiment.create --json '{"variants":[{"name":"control","note":"The Customer Lead’s current contract.","measure":"Conversations resolved with no hand-off that stay resolved for 7 days."},{"name":"refund-first","note":"Offer the refund before asking why, on Orders under $50."}]}' | jq -r '.experiment["$id"] | split("/") | last')
npx startup-builder do Experiment.start --json '{"experiment":"'"$EXPERIMENT"'"}'
```

MCP:

```json
{
  "name": "do",
  "arguments": {
    "code": "const experiment = await $.Experiment.create({ variants: [{ name: 'control', note: 'The Customer Lead’s current contract.', measure: 'Conversations resolved with no hand-off that stay resolved for 7 days.' }, { name: 'refund-first', note: 'Offer the refund before asking why, on Orders under $50.' }] })\nreturn await $.Experiment.start({ experiment: experiment.$id })"
  }
}
```

## Change it
- Stop the test if refunds double in either arm.
- Roll the winner out to the support seat.
- Test the reply length only.

## Under the hood
- Systems: [Experiments](/systems/experiments), [Support](/systems/support), [Evals](/agents/evals-system)
- Nouns: [Experiment](/business/nouns/experiment), [Variant](/business/nouns/variant), [Conversation](/business/nouns/conversation), [Decision](/business/nouns/decision)
- Events: [`Experiment.started`](/business/events/Experiment.started), [`Experiment.concluded`](/business/events/Experiment.concluded)

Wait for the volume. Below a few hundred Conversations per arm a test reads noise, so this wakes once the [Startup](/business/nouns/startup) has them. Until then, read ten transcripts a week and edit the contract by hand.