# Grade agent runs

Run your agent once per eval with rystic grid, each run on simulators of its own, grade every run PASS or FAIL with rystic grade and turn a good run into a regression eval.

`rystic grid <agent> -evals <dir>` runs your agent once per eval file. Each run, a cell, gets its own [simulators](https://www.rystic.ai/docs/concepts/glossary.md#simulator), loaded with the eval's [scenario](https://www.rystic.ai/docs/concepts/glossary.md#scenario), behind a strict [intercept](https://www.rystic.ai/docs/intercept.md) gateway of its own. The agent calls the vendor's real hosts unchanged, and no two cells share state.

When the agent stops, the cell is graded from the files it saved: PASS, FAIL or ERROR, with one reason. `rystic grade` grades a saved cell again without rerunning the agent. A coding agent does the same through `rystic mcp`: [Snapshots and test cases](https://www.rystic.ai/docs/concepts/snapshots-and-test-cases.md#test-cases).

## An eval file

An eval names the scenario each simulator loads, the job the agent gets and the checks that grade the run. This one asks for a Stripe customer:

```bash
rystic pull stripe-twin
mkdir -p evals add-customer
printf '%s\n' 'world:' '  stripe: design_studio' \
  'prompt: Add a customer named Ada Lovelace with the email ada@example.com.' \
  'checks:' \
  '  - forbid: {ref: stripe.after.state_customer, changed: [email]}' \
  "    desc: changed an existing customer's email" \
  '  - expect: {ref: stripe.after.state_customer, new: true, where: {email: ada@example.com}, count: 1}' \
  '    desc: no new customer for Ada' > evals/new_customer.yaml
```

| Key | What it holds |
|---|---|
| `world` | Required. Each simulator, named without `-twin`, mapped to the named scenario it loads. Checks name the simulators the same way |
| `prompt` | Required unless `inject` is set. The job, handed to the agent the way its `agent.yaml` says |
| `inject` | Per simulator, [timeline](https://www.rystic.ai/docs/timelines.md) events posted to `/_rystic/inject` when the cell starts, for an agent that waits for a request instead of reading a prompt |
| `request` | The id of the event that is the agent's request. A `{request: <field>}` matcher reads that event's call arguments |
| `checks` | Required. The checks, run in order |
| `allow_delete` | The deletions and destructive changes the task calls for, each `{ref, where}` on a `before` ref |

### Checks

| Check | Passes when |
|---|---|
| `expect: {ref, …}` | at least one row matches, or exactly `count` rows |
| `forbid: {ref, …}` | no row matches |
| `script: <path>` | the script, run with the cell directory as its one argument, prints no `FAIL <name> (<reason>)` line. It prints `PASS <name>` lines otherwise; a script that prints neither is an ERROR. The path is relative to the cell directory, so an eval's own script lives in a subfolder of the agent's `checks/`, named as `checks/<eval>/<file>` |

1. **`ref`** names the rows: `<simulator>.before.<register>` and `<simulator>.after.<register>` are a [register](https://www.rystic.ai/docs/concepts/glossary.md#register)'s rows before the agent started and after it stopped. `<simulator>.calls` is the [call log](https://www.rystic.ai/docs/concepts/observing.md), and `<simulator>.egress` the egress log.
2. **`new: true`** keeps the `after` rows whose key `before` lacks. **`changed: [fields]`** keeps the `after` rows where one of those fields differs from `before`.
3. **`where`** maps a field to a matcher, and a row matches when every field does. A dotted field reads into nested objects. A literal matches a deeply equal value; an object is one operator: `not`, `in`, `contains`, `exists` or `request`.
4. **`desc`** is the reason a failing check gives the cell. The first failing check's `desc` is the cell's reason, so list a specific wrong outcome as a `forbid` before the `expect` it would otherwise fail.
5. **Nothing destructive** runs before your checks. A row in `before` that is gone from `after`, or a change the simulator lists as destructive, fails the cell unless an `allow_delete` entry matches it.

## The agent

The agent is a folder carrying `agent.yaml`, or a command after `--`. This agent is one `curl` call to Stripe's real host, `https://api.stripe.com`:

```bash
printf '%s\n' 'name: add-customer' 'start: sh agent.sh' > add-customer/agent.yaml
printf '%s\n' 'curl -s --proto-default https api.stripe.com/v1/customers -u sk_test_agent: -d name="Ada Lovelace" -d email=ada@example.com' > add-customer/agent.sh
```

| Key | What it says |
|---|---|
| `name` | Required. Names the agent's row and cells: lowercase letters, digits, `.`, `_` and `-` |
| `start` | Required. The command, run with `sh -c` from the agent's folder |
| `done` | `exit`, the default, ends the cell when the process exits. `idle <duration>`, such as `idle 30s`, ends it once no simulator has served a request for that long |
| `events` | `poll`, the default, turns no push on. `socket` is Slack Socket Mode. `webhook <port> <path>` has the simulators deliver events to `http://127.0.0.1:<port><path>` |
| `allow` | Real hosts the agent may reach besides the simulators, such as its model's API. The gateway refuses every other host |
| `prompt` | `arg` replaces `{prompt}` in `start` with the eval's prompt, shell-quoted. `file` writes it to a file named by `RYSTIC_PROMPT_FILE`. Omitted, it is `arg` when `start` carries `{prompt}`, and `file` otherwise |

The agent keeps its own environment and keys. The cell adds the gateway's proxy and CA variables, as [`rystic exec`](https://www.rystic.ai/docs/intercept.md#the-environment) does, and the simulators accept any well-formed key.

## Run the grid

```bash
rystic grid add-customer -evals evals -out run1
# -> grid: add-customer × 1 evals, 2 at once → run1
# -> ...PASS  new_customer...
# -> 1 of 1 cells passed → run1
```

1. `-evals` takes a directory, whose `.yaml` files are the evals, or a comma-separated list of files. `-out` must not exist yet; it defaults to `grid-<timestamp>`.
2. `-j` sets how many cells run at once (default `2`). `-timeout` caps the agent's wall-clock time (default `5m`), and an agent still running at the cap fails as "never finished".
3. Each cell's clock moves one simulated second per wall second. `-pace 0` keeps it manual, and `-clock-step` then moves it that far each wall second. `-seed` pins every cell's random stream (default `1`).
4. A cell passes on its verdict, never on the agent's exit code. Any cell that is not PASS exits `7`.
5. `-ui` serves a grid page on a local port: rows go green or red as cells finish. **Run selected** and **Run failing** rerun those cells with the agent folder as it is now, and **Run all** reruns every cell. It serves until you interrupt it.

Each cell keeps its files under `<out>/<eval>/`: `cell.json`, the `eval.yaml` it ran, `agent.log`, `verdict.json` and per simulator `twins/<name>/before.json`, `after.json`, `calls.json`, `egress.json`, `world.json` and `twin.log`. `<out>/ledger.jsonl` holds one row per run of a cell, with its verdict, reason and checks digest.

## Grade a saved run

`rystic grade` takes cell directories or a grid's `-out` directory, and grades each cell from its saved files, never a live simulator:

```bash
rystic grade run1
# -> PASS  new_customer
# -> 1 of 1 cells passed
jq -c '{verdict, reason}' run1/new_customer/verdict.json
# -> {"verdict":"PASS","reason":""}
```

It rewrites each cell's `verdict.json`, exits `7` when a cell is not PASS and `1` when one cannot be graded. ERROR means the infrastructure failed, not the agent.

### Your own checks: checks/

Checks your evals all share go directly in the agent folder's `checks/`: executable scripts and YAML files holding a `checks` list and an optional `allow_delete`. Each one runs in every cell; a subfolder runs only where a check names it. Grid copies the folder into each cell, and they run after the eval's own checks. A script a YAML file names is relative to `checks/`:

```bash
mkdir -p add-customer/checks
printf '%s\n' 'checks:' \
  '  - forbid: {ref: stripe.after.state_customer, new: true, where: {email: {contains: example.com}}}' \
  '    desc: created a customer with a placeholder email' > add-customer/checks/real-emails.yaml
rystic grid add-customer -evals evals -out run2; echo "exit $?"
# -> grid: add-customer × 1 evals, 2 at once → run2
# -> ...FAIL  new_customer ... created a customer with a placeholder email...
# -> 0 of 1 cells passed → run2
# -> each cell's verdict.json, agent.log and twins/*/twin.log are under run2
# -> exit 7
```

## Draft an eval from a good run

`rystic grade --draft <cell>` grades nothing. It prints an eval drawn from one finished cell, with the cell's scenarios, prompt and injected events:

```bash
rystic grade --draft run1/new_customer > evals/new_customer_draft.yaml
head -3 evals/new_customer_draft.yaml
# -> # DRAFT: rystic grade --draft drew these checks from one recorded run.
# -> # Agent: add-customer. Eval: new_customer.
# -> # They expect every row the run created or changed, forbid every other
```

Its checks expect each row the run created or changed, and forbid any other new row, changed field, caller or method. `allow_delete` names each row the run deleted or changed destructively. Ids and times the run generated stay out. A run that never finished, or that would fail its own draft, prints no draft and exits with an error. Trim the checks to what the task requires before you keep the file.

## Command evaluators

A simulator with an evaluation engine, today [GitHub](https://www.rystic.ai/docs/simulators/github-pulls/features/ci-evaluation.md) for its workflow runs, hands the commands it needs run to `rystic evaluator`. It claims each one, runs it in a digest-pinned container with no network and reports the outcome:

```bash
rystic pull github-pulls-twin
rystic run github-pulls-twin -d --name ci --port 0
rystic evaluator --instance ci --once 2>&1
# -> ...evaluator: completed 0
rystic stop ci
```

1. It runs until `Ctrl-C`; `--once` runs what is waiting, then exits. `--url <base>` names a simulator that `rystic ps` does not list.
2. A command exits `0` to pass, `1` to fail and `78` to decline the input, or writes its outcome to the file `$EVAL_RESULT` names. Anything else is an infrastructure error.
3. `--runner` is `docker` (the default) or `podman`. `--runner local --trusted-local` runs local commands with host access, for trusted code only. `--concurrency` sets how many run at once (default `2`).

---

Source: https://www.rystic.ai/docs/agent-evals · Markdown: https://www.rystic.ai/docs/agent-evals.md