Evaluator · StartupBuilder

Properties

Property	Type	Required	Description
`rubric`	`text`	required	What it grades against. An agent never varies its own Evaluators. Never randomized.
`integration`	Integration		The Integration whose outside run returns the verdict, such as a CI job; empty when the platform runs it.
`required`	`boolean`		True when `Change.merge` refuses until this Evaluator passes on the Change’s head. Never randomized.

Inherited from Function: 12 properties

Property	Type	Required	Description
`kind`	`string`	required	One of `code`, `generative`, `agentic`, or `human`.
`version`	`string`		The Version an Action runs, kept with the Action.
`cascade`	`string[]`		Kinds tried in order when one declines, such as code, then a Human.
`instructions`	`text`		For a generative or agentic Function, what it does and how.
`model`	`string`		For a generative or agentic Function, the model it runs on.
`tools`	list of Verb		For an agentic Function, the Verbs it may call; Permissions and Limits apply to each.
`maxSteps`	`integer`		For an agentic Function, the most steps one Action may take.
`budget`	`money`		The most one Action may spend; the Mandate’s Limits apply as well.
`role`	Role		For a human Function, the Role its Tasks go to.
`form`	`json`		For a human Function, the form; generated from the Verb’s inputs and outputs when absent.
`sla`	`string`		For a human Function, how long a Task may wait, as an ISO 8601 duration.
`escalateTo`	Role		For a human Function, the Role a Task moves to when the service level passes.

Object

JSON

{
  "$id": "https://api.sb/ledgerlark.com/evaluators/evaluator_1",
  "$type": "Evaluator",
  "kind": "code",
  "version": "Example version",
  "cascade": [
    "Example cascade"
  ],
  "instructions": "Example instructions",
  "model": "Example model",
  "tools": [
    "verb_1"
  ],
  "maxSteps": 1,
  "budget": {
    "amount": 400,
    "currency": "USD"
  },
  "role": "role_1",
  "form": {},
  "sla": "PT4H",
  "escalateTo": "role_1",
  "rubric": "Example rubric",
  "integration": "integration_1",
  "required": true
}

Relationships

An Evaluator grades many Verbs: the Verbs whose results the Evaluator scores against its rubric before they count as done. grades, inverse gradedBy.
An Evaluator verifies many Outcome Contracts: Evaluators grade delivered work against an Outcome Contract’s done state before the Customer accepts it. verifies, inverse verifiedBy.

Verbs

createCreates an Evaluator.
POST /{namespace}/evaluatorsdoEvaluator.created
getReturns one Evaluator by id.
GET /{namespace}/evaluators/{id}readNo Event
listReturns Evaluators, newest first, one page at a time.
GET /{namespace}/evaluatorsreadNo Event
updateChanges the given properties of an Evaluator.
PATCH /{namespace}/evaluators/{id}doEvaluator.updated
deleteDeletes an Evaluator; its history is kept.
DELETE /{namespace}/evaluators/{id}doEvaluator.deleted

Permissions

Which Access level may call each Verb. One-way Verbs wait their delay window.

Operation	`owner`	`admin`	`member`	`viewer`
`Evaluator.create`	Yes	Yes	Yes	No
`Evaluator.get`	Yes	Yes	Yes	Yes
`Evaluator.list`	Yes	Yes	Yes	Yes
`Evaluator.update`	Yes	Yes	Yes	No
`Evaluator.delete`	Yes	Yes	Yes	No

Call it

Text

Add an evaluator to PatentPad.

Example

An Evaluator grades each support answer against a rubric of accuracy, tone, and policy, and returns pass or fail with the reason.

PropertiesCopy Anchor Link

ObjectCopy Anchor Link

RelationshipsCopy Anchor Link

VerbsCopy Anchor Link

PermissionsCopy Anchor Link

Call itCopy Anchor Link

ExampleCopy Anchor Link