Agent definition evals
Every Change to an instruction, Tool, Skill, model Policy or model Version passes its Evaluator regression suite before merge.
Satisfies
| Criterion | Text |
|---|---|
| SOC2 CC8.1 | The entity authorizes, designs, develops, configures, documents, tests, approves and implements changes to infrastructure, data, software and procedures. |
| ISO27001 A.8.29 | Security testing processes are defined and implemented in the development life of every system. |
Implemented by
Test
| Step | Rule |
|---|---|
| Kind | Preventive: it stops the failure before it happens. |
| Select | Every Change.merged Event whose target is an instruction, Tool, Skill, Policy or model Version. |
| Assert | Each cites a passing Evaluator verdict showing the new Version no worse than the one it replaces. |
Owner
Example
A Change rewrites the support Agent's instructions for a recruiting SaaS. The Evaluator suite replays 400 past Conversations against the new Version, which scores 0.2 points below the old one, so the Change cannot merge.