# Evaluator

URL: /business/nouns/evaluator · Id: `noun:evaluator`

> An Evaluator is a versioned Function that grades an output against a rubric and returns a verdict.

An Evaluator is a versioned [Function](/business/nouns/function) that grades an output against a rubric and returns a verdict.

## Properties
| Property | Type | Required | Description |
| --- | --- | --- | --- |
| `rubric` | `text` | Yes | What it grades against. An agent never varies its own Evaluators. Never randomized. |
| `integration` | [Integration](/business/nouns/integration) | No | The Integration whose outside run returns the verdict, such as a CI job; empty when the platform runs it. |
| `required` | `boolean` | No | True when `Change.merge` refuses until this Evaluator passes on the Change’s head. Never randomized. |

<details>
<summary>Inherited from [Function](/business/nouns/function): 12 properties</summary>

| Property | Type | Required | Description |
| --- | --- | --- | --- |
| `kind` | `string` | Yes | One of `code`, `generative`, `agentic`, or `human`. |
| `version` | `string` | No | The Version an Action runs, kept with the Action. |
| `cascade` | `string[]` | No | Kinds tried in order when one declines, such as code, then a Human. |
| `instructions` | `text` | No | For a generative or agentic Function, what it does and how. |
| `model` | `string` | No | For a generative or agentic Function, the model it runs on. |
| `tools` | list of [Verb](/business/nouns/verb) | No | For an agentic Function, the Verbs it may call; Permissions and Limits apply to each. |
| `maxSteps` | `integer` | No | For an agentic Function, the most steps one Action may take. |
| `budget` | `money` | No | The most one Action may spend; the Mandate’s Limits apply as well. |
| `role` | [Role](/business/nouns/role) | No | For a human Function, the Role its Tasks go to. |
| `form` | `json` | No | For a human Function, the form; generated from the Verb’s inputs and outputs when absent. |
| `sla` | `string` | No | For a human Function, how long a Task may wait, as an ISO 8601 duration. |
| `escalateTo` | [Role](/business/nouns/role) | No | For a human Function, the Role a Task moves to when the service level passes. |

</details>

## Object
```json
{
  "$id": "https://api.sb/ledgerlark.com/evaluators/evaluator_1",
  "$type": "Evaluator",
  "kind": "code",
  "version": "Example version",
  "cascade": [
    "Example cascade"
  ],
  "instructions": "Example instructions",
  "model": "Example model",
  "tools": [
    "verb_1"
  ],
  "maxSteps": 1,
  "budget": {
    "amount": 400,
    "currency": "USD"
  },
  "role": "role_1",
  "form": {},
  "sla": "PT4H",
  "escalateTo": "role_1",
  "rubric": "Example rubric",
  "integration": "integration_1",
  "required": true
}
```

## Relationships
- <a id="rel-grades"></a>An Evaluator **grades** many [Verbs](/business/nouns/verb): the [Verbs](/business/nouns/verb) whose results the Evaluator scores against its rubric before they count as done. `grades`, inverse [`gradedBy`](/business/nouns/verb#rel-graded-by).
- <a id="rel-verifies"></a>An Evaluator **verifies** many [Outcome Contracts](/business/nouns/outcome-contract): Evaluators grade delivered work against an [Outcome Contract](/business/nouns/outcome-contract)’s done state before the [Customer](/business/nouns/customer) accepts it. `verifies`, inverse [`verifiedBy`](/business/nouns/outcome-contract#rel-verified-by).

## Verbs
| Verb | Description | Subject | Event | Class |
| --- | --- | --- | --- | --- |
| <a id="create"></a>[Evaluator.create](/business/nouns/evaluator/create) | Creates an Evaluator. Inherited from [Noun](/business/nouns/noun). | None. | `Evaluator.created` | Two-way: `try`, `send`, `do` |
| <a id="get"></a>[Evaluator.get](/business/nouns/evaluator/get) | Returns one Evaluator by id. Inherited from [Noun](/business/nouns/noun). | None. | None. | Read: `try`, `do` |
| <a id="list"></a>[Evaluator.list](/business/nouns/evaluator/list) | Returns Evaluators, newest first, one page at a time. Inherited from [Noun](/business/nouns/noun). | None. | None. | Read: `try`, `do` |
| <a id="update"></a>[Evaluator.update](/business/nouns/evaluator/update) | Changes the given properties of an Evaluator. Inherited from [Noun](/business/nouns/noun). | None. | `Evaluator.updated` | Two-way: `try`, `send`, `do` |
| <a id="delete"></a>[Evaluator.delete](/business/nouns/evaluator/delete) | Deletes an Evaluator; its history is kept. Inherited from [Noun](/business/nouns/noun). | None. | `Evaluator.deleted` | One-way: `send`, `do` |

## Permissions
Which Access level may call each Verb. One-way Verbs wait their delay window.

| Operation | `owner` | `admin` | `member` | `viewer` |
| --- | --- | --- | --- | --- |
| `Evaluator.create` | Yes | Yes | Yes | No |
| `Evaluator.get` | Yes | Yes | Yes | Yes |
| `Evaluator.list` | Yes | Yes | Yes | Yes |
| `Evaluator.update` | Yes | Yes | Yes | No |
| `Evaluator.delete` | Yes | Yes | Yes | No |

## Call it

**Ask**

```text
Add an evaluator to PatentPad.
```

**API**

```bash
curl -X POST https://api.sb/patentpad.com/evaluators \
  -H "Authorization: Bearer $STARTUP_BUILDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind":"code","rubric":"Example rubric"}'
```

```json
{
  "api": "https://api.sb/patentpad.com",
  "evaluator": {
    "$id": "https://api.sb/patentpad.com/evaluators/evaluator_1",
    "$type": "Evaluator",
    "kind": "code",
    "rubric": "Example rubric"
  },
  "action": {
    "$id": "https://api.sb/patentpad.com/actions/action_1",
    "operation": "Evaluator.create",
    "status": "done"
  },
  "links": {
    "self": "https://api.sb/patentpad.com/actions/action_1"
  }
}
```

**SDK**

```ts
import { $ } from 'startup-builder'

const evaluator = await $.Evaluator.create({ kind: 'code', rubric: 'Example rubric' })
```

**CLI**

```bash
npx startup-builder do Evaluator.create --json '{"kind":"code","rubric":"Example rubric"}'
```

**MCP**

```json
{
  "name": "do",
  "arguments": {
    "code": "return await $.Evaluator.create({ kind: 'code', rubric: 'Example rubric' })"
  }
}
```

## Example
An Evaluator grades each support answer against a rubric of accuracy, tone, and policy, and returns pass or fail with the reason.