research · safety
A safety CLI for agent runtimes
A command-line approach to choosing checks, finding failures and keeping claims within the evidence, with interactive examples.
By Ali Madad · Published
A simulated conversation ends without an error. The report calls it a pass. But no behavioral requirement was checked. What evidence supports the result?
This is a useful starting point for a safety CLI: make each check's purpose clear, then keep the claim within what it measured. A completed run can provide a transcript for review. Calling it a pass requires a defined requirement and evidence that the requirement held.
A command-line interface gives developers and coding agents a common way to select checks and inspect results. Existing test runners retain their work. The CLI brings their commands into a repeatable review process.
Scope of this article. This describes an internal development method. The CLI has not been released as a public package. Command examples use pnpm quality. The interactive examples use synthetic records; they illustrate concepts without executing an evaluator or calling a model.
A completed run still needs a check
Suppose a user asks an assistant for one check-in. A contract could require exactly one check-in to be queued. The stored result can establish whether that requirement held. Reply quality and user benefit need other evidence.
Select a record below. The selected record changes what the evidence supports.
What does this run establish?
Synthetic records. These controls do not run an evaluator or call a model.
- Execution
- Complete
- Contract result
- NOT GRADED
The conversation ended. No contract checks were defined.
There is a transcript to inspect. No checked requirement supports a pass.
With no contract checks, the result is NOT GRADED. One passing check supports a claim about that requirement in that run. A second queued check-in breaks it. A provider error needs its own status because the attempt did not complete as intended.
A reviewer may still need to read the delivered reply. Keep that pending review visible beside the contract result. Otherwise, a narrow mechanical pass can look like a judgment about the whole interaction.
Organize commands around questions
The CLI should help a person or coding agent choose evidence for a decision. A change to event ordering needs different checks from a change to a model question. The command names make those purposes visible.
| Command | Question it addresses | Limit |
|---|---|---|
plan | Which checks does this change need? | A plan prepares work; it does not provide test evidence. |
architecture | Does each state and effect have a clear owner? Is custom orchestration needed? | Automatic checks cover declared rules. Design judgment needs code review. |
formal | Do the modeled rules hold across the event orders explored? | The result depends on the model's bounds and assumptions. |
check / build | Do regression checks pass? Can the application build? | These checks do not establish the quality of a model's answers. |
replay | What does the application commit and deliver in a selected scenario? | Contract results apply to the cases tested. |
judge / calibration | How do judge answers compare with reviewed labels? | Agreement does not establish that the reference labels are correct. |
read / compare | What failed? Can two runs support a comparison? | Missing or incompatible evidence must remain visible. |
The command layer can stay small. It forwards options to existing tools and preserves their failures. Tests, scenario data and run records can keep their current owners. A single entry point does not require every check to live in the same directory.
pnpm quality plan
pnpm quality architecture
pnpm quality check
Each result needs a scope. A successful lint command can confirm import rules while leaving design review open. Combining both into a single safety score would hide that distinction.
Find the state machines hidden in the app
Architecture review can start with a reusable prompt:
Find all explicit and implicit state machines in the app. Look beyond status enums. Trace state across flags, nullable fields, timestamps, tasks and callbacks. Include states that exist only in control flow. For each machine, name its owner and legal transitions. Check what happens after a duplicate event, interruption or restart. Then ask whether a supported framework primitive can replace custom code while preserving the same guarantee.
A queued action may have a database record and a scheduled task. Their presence alone does not prove duplication. The reviewer must identify what each part guarantees and trace the effects of removing it.
Check the installed framework version before proposing a replacement. Follow callers and stored formats as well as imports. A later permission check may still be necessary because permission can change while work waits.
Record each proposed removal with the rule it preserves and a test that would catch a regression. Leave uncertain findings open. A CLI can prepare this review and run declared checks; deciding whether the design fits its purpose requires judgment.
Check event order explicitly
Consider a generic assistant that saves notes. An answer reads a note before a removal request arrives. If the answer later writes the old text back into context, clearing the stored note may not achieve the intended removal.
This generic illustration changes the order of those operations.
Change the order of a note removal
Simplified model illustration. This control does not execute TLA+ or change saved data.
- An answer in progress reads the saved note.
- The removal clears the note and resets the context.
- The old work finishes and places text from the note back into context.
- A later request can see the removed information.
Counterexample: removed information returns to future context.
In the unsafe sequence, old work finishes after the reset and restores the information. Waiting for the relevant work before removal completes prevents that result in the illustrated sequence. A real implementation needs a rule that covers its own possible event orders and failure paths.
TLA+ is a language for specifying systems. Its TLC model checker can explore a finite model of a concurrent process. That can expose an ordering failure which a single successful execution would miss.
A useful formal check includes a known broken variant. The expected counterexample shows that the check can detect the failure it names. An unrelated error must not count as that counterexample. Keep the original trace so a reviewer can follow the steps and inspect the state values.
pnpm quality formal --since "$REVIEWED_REVISION"
The source comparison serves another purpose: it identifies changes since a prior model-to-code review. A model can keep passing after the code has changed enough to make it a poor description. Record the reviewed revision and explain the assumptions. Runtime tests must still exercise the corresponding behavior in code.
Keep discovery separate from acceptance
A simulation can generate a user's next message. A perturbation can change spelling or punctuation while retaining the intended request. Red-team scenarios can target a suspected failure. These methods help find cases that deserve investigation.
Acceptance evidence needs stable conditions. Fix the selected cases and scoring rules before comparing a baseline with a candidate. A discovered case becomes a regression after someone establishes the expected behavior. If the grader changes, score both sides with the same revised grader.
Offline evaluation uses isolated application state. It can still call paid model APIs. Online checks exercise a deployed service, while monitoring observes production signals. The method should state which setting a run uses and what access it needs.
# Choose a discovery case and approve its request and cost limits first.
pnpm quality replay --dataset dev --case "$CASE_ID" \
--out results/replay/discovery-example \
--max-model-requests "$REQUEST_CAP" \
--max-estimated-cost-usd "$COST_CAP"
Make the unit of each limit explicit. A limit per case or repeat does not cap the whole experiment. Keep each attempt in a separate output directory, including attempts that fail.
Use failures to improve the next check
A useful improvement loop starts with the first supported cause of a failure. The problem might be in the scenario, the application, the provider, or the check itself. Preserve the failed attempt before changing any of them.
Reduce the failure to a case that retains the relevant behavior. Write a regression which fails before the fix, then rerun it after one targeted change. Follow with the affected checks. A broader acceptance run can then test whether the fix damaged another selected behavior.
When the disagreement concerns meaning, a mechanical regression is only part of the work. A judge may agree with a wrong reference label. Repeated model agreement cannot replace independent expert review of consequential labels.
The CLI supports this loop by keeping commands repeatable and evidence available. Permission to change a live system remains a separate decision. A finding can justify a proposed fix without authorizing an automatic release.
Put the method in daily work
Developers and coding agents use the CLI before a change to select checks, during the change to investigate failures, and before handoff to collect evidence. Eval reviewers inspect the delivered results. Release operators check which claims remain unresolved before making a release decision.
Start with the tools the team already runs. Give each command a clear question and an explicit failure result. Add a check when it protects a named requirement. Keep the evidence accessible enough that another person can follow the failure, the proposed cause and the recheck.
Method note: all examples on this page are synthetic. The article describes a general verification approach. It makes no claim about production safety, independent judge accuracy or improved user outcomes.