Evals
Evals grades outputs with versioned Evaluators, gates every Change before merge, and keeps each Worker’s track record.
Evals grades outputs with versioned Evaluators, gates every Change before it merges, and keeps each Worker’s track record. An Evaluator is code, a model, an Agent or a Human, and returns a verdict, never an Approval.
Summary
| Part | This System |
|---|---|
| Group | Moves to Agents |
| Status | Beta: it works, and its Verbs may still change |
| Shelf | Front: offered from a Startup’s first days |
You can ask:
- “How good are the support agent’s answers?”
- “Grade this agent’s last 50 replies”
- “Block changes that make answers worse”
What you get
| Part | This System |
|---|---|
| Starts | On for every Startup at creation, with nothing to install. |
| Built from | Functions, AI gateway, Event streams |
Nouns it owns
| Noun | Definition |
|---|---|
| Evaluator | An Evaluator is a versioned Function that grades an output against a rubric and returns a verdict. |
Verbs
| Verb | Object | Approval | Event |
|---|---|---|---|
| Evaluator.create | Evaluator | None | Evaluator.created |
| Action.evaluate | Action | None | Action.evaluated |
Events
| Direction | Event | How |
|---|---|---|
| Emits | Evaluator.created | By Evaluator.create |
| Emits | Action.evaluated | By Action.evaluate |
States
None.
Who operates it
| Part | This System |
|---|---|
| Accountable | Quality Lead |
| Team | Quality Team |
| Seat | A role agent holds it until a Human is seated. |
| Autonomy | supervised |
| Also calls its Verbs | Reviewer, Tester |
Approvals and limits
None.
Processes that use it
| Process | How it uses it |
|---|---|
| Verification | A step calls Action.evaluate |
| Hiring | A step calls Action.evaluate |
Rail underneath
None.
Views and dashboards
| View | Shows | Metrics |
|---|---|---|
| The Quality Lead’s view | The Metrics its accountable Role answers for. | Escaped defects |
Access
What each Access level may call here.
| Access level | May call |
|---|---|
owner | Every Verb (6) |
admin | Every Verb (6) |
member | Every Verb (6) |
viewer | Read only (2 of 6) |
Import and export
None.
Fees and pricing
None.
Availability and limits
| Part | This System |
|---|---|
| Status | Beta: it works, and its Verbs may still change |
Compliance notes
Controls it carries:
Related Systems
None.
Example
An E2E managed service for customer support rewrites its refund-reply prompt. Evals replays 400 past Conversations through it; the new prompt scores no worse on accuracy or tone, so the Change merges. The Agent’s track record, which earns it wider Mandates, updates with the result.