Quality playbook
How the Quality Team checks every Change, every agent definition, every night and everything that leaves a Startup, with verdicts instead of Approvals.
Quality in a Startup is a set of checks that run inside every Process, not a gate at the end. The Quality Team writes the checks, runs them on every Change and every night, and reviews what leaves the Startup.
Verdicts, not approvals
An Approval lets an Action proceed on someone’s say-so. A verdict states whether checks passed, with the Evidence. The Quality Team returns only verdicts, so no Change waits on anyone’s opinion.
Every check is deterministic where it can be: a field has its shape, a link resolves, a count meets its threshold. Two runs over the same state return the same verdict. Where a call needs taste, an Evaluator scores it against a rubric, and the score becomes Evidence.
A Task closes only when its checks pass, never on a Worker’s word. When an Agent reports a Page published, the check fetches the live Page and counts its links before the Task closes.
Every Change, before it merges
Change verdict runs when a Change is created. It runs the Change’s checks and replays known-good runs against it. It also sends a synthetic paying stranger through signup, payment and first value.
Take a Marketplace for certified translators. A Change makes a tax field required at checkout in every country, so a synthetic stranger from a country without that tax cannot pay. The Tester records the fail with the Action attached, and the Change waits for a fix.
A pass lets the Change merge, and the Engineering Team ships it by traffic share. Quality never deploys; it only says whether the Change is fit.
Agent definitions are tested like code
Everything an Agent runs is a versioned definition: its instructions, Skills, Tools and model Policies. Each has an eval suite of cases drawn from real runs. A new variant passes that suite offline, no worse than the current Version, before any traffic sees it.
Online, the variant runs as an Experiment on the one Metric that matters, with guardrails that stop it on harm. The winner compiles into the definition. A loser concludes and stays on record.
Take a support Agent for a language-tutoring app. A new instruction set passes 231 of 240 cases, against 228 for the current one. At 10% of Conversations it resolves 4 points more without a handoff, so it ships to all.
The same record earns autonomy. A Role agent moves from assist to supervised to full when its eval track record holds at 95% or better. It moves back down on its own when the record slips.
Reviewing what leaves the Startup
A Message cannot be unsent and money cannot be unspent. The Reviewer therefore checks every outbound Message and every money move at medium stakes or above.
Stakes set the delay window. By default low proceeds at once, medium waits one business hour, and high waits one business day. Windows are Settings that inherit down from the Studio, and a Startup can only tighten them.
Guards run first, in code. Evaluators run next, wherever the Policy enables them. A failed review cancels or redirects the Action with the reason recorded, and the Reviewer may raise a tier, never lower one.
Coverage matters more than cleverness. Reviewing only risky-looking Messages lets attacks through, and so does reviewing Messages but not refunds.
The nightly run
At 03:00 the Nightly quality run re-runs every check across the Startup. It replays known-good Actions on every Interface and runs the synthetic stranger again. Each regression becomes an Issue with the Action that reproduces it.
Interface parity is one of those checks. Every Verb behaves the same through the API, CLI, MCP and SDK, and the nightly run proves it.
Next
- Quality Team: who staffs it, what runs when, and what it measures.
- Engineering playbook: how a passing Change reaches all traffic.
- Keep the winner: how a winning variant becomes the definition.