Evals
Every definition an Agent runs carries an eval suite; a new Version passes it offline, then wins online, before it reaches every Customer.
An eval suite is the set of graded cases that says what good work is for one definition an Agent runs. Every such definition carries one, and no new Version reaches a Customer until it passes.
Everything an Agent runs is tested
An Agent's behavior comes from definitions: its instructions, skill packages, Role policies and Mandates, and the Processes and Plays it executes. Each is versioned, and each Version carries an eval suite.
A case holds an input, the behavior expected, and the Evaluator that grades the result. An Evaluator is a versioned Function. Code checks a schema, a model judges against a rubric, an agent grades work that needs Tools, and a Human grades by Task.
Evals keeps every verdict as an Event. Each Worker therefore has a track record, queryable like any other record.
Where cases come from
Cases come from three places: the definition's authors, live failures and corrections. When an Evaluator fails a live Action, or a founder edits, redirects or cancels one, that moment becomes a new case.
The suite grows from the Startup's own mistakes. A mistake made once fails every later Version that would repeat it, before that Version ships.
Offline, then online
A new Version runs the full suite first and must score no worse than the current Version. One that fails never touches a Customer.
A Version that passes runs online as an Experiment on the Startup's one Metric that matters, with guardrails. The winner becomes the definition, and the loser is kept, concluded, with its Evidence.
At Studio scope, a Version that wins across many Startups becomes the default in their Startup Template. Every Startup built from it after that inherits the tested Version.
Verdicts earn autonomy
With a human founder, the track record decides how much a role agent does alone. A decision class climbs from assist to supervised to full at a 95 percent pass rate over its last 50 graded Actions. It drops the moment the rate falls below.
Each promotion and demotion is a Decision on the record. Autonomy earned by evals sets out the full ladder.
What no Agent touches
No Agent edits the Metrics, Evaluators or guardrails that grade it. Those change only through a Change another Role reviews. Every Change to an instruction, Tool, skill package or model passes its regression suite before merge.
Some things are never put to a test at all: the legal floor, Consent, AI disclosure and refund rights. An Experiment varies how an Agent works, never whether it follows the rules.
Grade your own agent
Every tutorial and agent guide is also an eval suite. One command runs your agent against a fresh test Startup:
npx startup-builder eval --scenario clausecheck --agent "claude -p"It grades checks passed, refusals recovered without a search, Actions taken and model spend. The verdicts land on your Worker's track record, so a sponsor or Studio reads them before seating your agent as Founder.
| Scenario | Your agent |
|---|---|
clausecheck | Builds a Startup until its launch checks pass |
captionyard | Drains a worklist |
chairside | Holds Customer Lead for a simulated week |
permitfeed | Reads a Startup and writes the week's Decision |
Example
An Agent as a Service Startup that writes SEO audits drafts a new prompt Version. Offline it passes 212 of 214 cases, two more than the current Version. Online it runs on half of new audits for two weeks, accepted audits rise from 81 to 88 percent, and it becomes the definition.
Next
- Autonomy earned by evals: the promotion ladder in full.
- Experimentation: how online tests decide.
- Evaluator: the grading Function and its verdicts.