# Evals

URL: /agents/evals · Id: `page:agents/evals`

> Every definition an Agent runs carries an eval suite; a new Version passes it offline, then wins online, before it reaches every Customer.

An eval suite is the set of graded cases that says what good work is for one definition an [Agent](/business/nouns/agent) runs. Every such definition carries one, and no new [Version](/business/nouns/version) reaches a [Customer](/business/nouns/customer) until it passes.

## Everything an Agent runs is tested
An Agent's behavior comes from definitions: its instructions, skill packages, [Role](/business/nouns/role) policies and [Mandates](/business/nouns/mandate), and the [Processes](/business/nouns/process) and [Plays](/business/nouns/play) it executes. Each is versioned, and each Version carries an eval suite.

A case holds an input, the behavior expected, and the [Evaluator](/business/nouns/evaluator) that grades the result. An Evaluator is a versioned [Function](/business/nouns/function). Code checks a schema, a model judges against a rubric, an agent grades work that needs [Tools](/business/nouns/tool), and a [Human](/business/nouns/human) grades by [Task](/business/nouns/task).

[Evals](/agents/evals-system) keeps every verdict as an [Event](/business/nouns/event). Each [Worker](/business/nouns/worker) therefore has a track record, queryable like any other record.

## Where cases come from
Cases come from three places: the definition's authors, live failures and corrections. When an Evaluator fails a live [Action](/business/nouns/action), or a founder edits, redirects or cancels one, that moment becomes a new case.

The suite grows from the [Startup](/business/nouns/startup)'s own mistakes. A mistake made once fails every later Version that would repeat it, before that Version ships.

## Offline, then online
A new Version runs the full suite first and must score no worse than the current Version. One that fails never touches a Customer.

A Version that passes runs online as an [Experiment](/business/nouns/experiment) on the Startup's one [Metric](/business/nouns/metric) that matters, with guardrails. The winner becomes the definition, and the loser is kept, concluded, with its [Evidence](/business/nouns/evidence).

At [Studio](/business/nouns/studio) scope, a Version that wins across many Startups becomes the default in their [Startup Template](/business/nouns/startup-template). Every Startup built from it after that inherits the tested Version.

## Verdicts earn autonomy
With a human founder, the track record decides how much a role agent does alone. A decision class climbs from `assist` to `supervised` to `full` at a 95 percent pass rate over its last 50 graded Actions. It drops the moment the rate falls below.

Each promotion and demotion is a [Decision](/business/nouns/decision) on the record. [Autonomy earned by evals](/agents/autonomy-levels/evals) sets out the full ladder.

## What no Agent touches
No Agent edits the Metrics, Evaluators or guardrails that grade it. Those change only through a [Change](/business/nouns/change) another Role reviews. Every Change to an instruction, Tool, skill package or model passes its regression suite before merge.

Some things are never put to a test at all: the legal floor, [Consent](/business/nouns/consent), AI disclosure and refund rights. An Experiment varies how an Agent works, never whether it follows the rules.

## Grade your own agent
Every tutorial and agent guide is also an eval suite. One command runs your agent against a fresh test Startup:

```bash
npx startup-builder eval --scenario clausecheck --agent "claude -p"
```

It grades checks passed, refusals recovered without a search, Actions taken and model spend. The verdicts land on your Worker's track record, so a sponsor or Studio reads them before seating your agent as [Founder](/agents/founding/founder).

| Scenario | Your agent |
| --- | --- |
| `clausecheck` | Builds a Startup until its launch checks pass |
| `captionyard` | Drains a worklist |
| `chairside` | Holds [Customer Lead](/agents/customer/customer-lead) for a simulated week |
| `permitfeed` | Reads a Startup and writes the week's Decision |

## Example
An [Agent as a Service](/products/templates/agent-as-a-service) Startup that writes SEO audits drafts a new prompt Version. Offline it passes 212 of 214 cases, two more than the current Version. Online it runs on half of new audits for two weeks, accepted audits rise from 81 to 88 percent, and it becomes the definition.

## Next
- [Autonomy earned by evals](/agents/autonomy-levels/evals): the promotion ladder in full.
- [Experimentation](/code/builders/startup/experimentation): how online tests decide.
- [Evaluator](/business/nouns/evaluator): the grading Function and its verdicts.