Grade agent runs

rystic grid <agent> -evals <dir> runs your agent once per eval file. Each run, a cell, gets its own simulators, loaded with the eval’s scenario, behind a strict intercept gateway of its own. The agent calls the vendor’s real hosts unchanged, and no two cells share state.

When the agent stops, the cell is graded from the files it saved: PASS, FAIL or ERROR, with one reason. rystic grade grades a saved cell again without rerunning the agent. A coding agent does the same through rystic mcp: Snapshots and test cases.

An eval file

An eval names the scenario each simulator loads, the job the agent gets and the checks that grade the run. This one asks for a Stripe customer:

rystic pull stripe-twin
mkdir -p evals add-customer
printf '%s\n' 'world:' '  stripe: design_studio' \
  'prompt: Add a customer named Ada Lovelace with the email ada@example.com.' \
  'checks:' \
  '  - forbid: {ref: stripe.after.state_customer, changed: [email]}' \
  "    desc: changed an existing customer's email" \
  '  - expect: {ref: stripe.after.state_customer, new: true, where: {email: ada@example.com}, count: 1}' \
  '    desc: no new customer for Ada' > evals/new_customer.yaml

Checks

1. ref names the rows: <simulator>.before.<register> and <simulator>.after.<register> are a register’s rows before the agent started and after it stopped. <simulator>.calls is the call log, and <simulator>.egress the egress log.
2. new: true keeps the after rows whose key before lacks. changed: [fields] keeps the after rows where one of those fields differs from before.
3. where maps a field to a matcher, and a row matches when every field does. A dotted field reads into nested objects. A literal matches a deeply equal value; an object is one operator: not, in, contains, exists or request.
4. desc is the reason a failing check gives the cell. The first failing check’s desc is the cell’s reason, so list a specific wrong outcome as a forbid before the expect it would otherwise fail.
5. Nothing destructive runs before your checks. A row in before that is gone from after, or a change the simulator lists as destructive, fails the cell unless an allow_delete entry matches it.

The agent

The agent is a folder carrying agent.yaml, or a command after --. This agent is one curl call to Stripe’s real host, https://api.stripe.com:

printf '%s\n' 'name: add-customer' 'start: sh agent.sh' > add-customer/agent.yaml
printf '%s\n' 'curl -s --proto-default https api.stripe.com/v1/customers -u sk_test_agent: -d name="Ada Lovelace" -d email=ada@example.com' > add-customer/agent.sh

The agent keeps its own environment and keys. The cell adds the gateway’s proxy and CA variables, as rystic exec does, and the simulators accept any well-formed key.

Run the grid

rystic grid add-customer -evals evals -out run1
# -> grid: add-customer × 1 evals, 2 at once → run1
# -> ...PASS  new_customer...
# -> 1 of 1 cells passed → run1

1. -evals takes a directory, whose .yaml files are the evals, or a comma-separated list of files. -out must not exist yet; it defaults to grid-<timestamp>.
2. -j sets how many cells run at once (default 2). -timeout caps the agent’s wall-clock time (default 5m), and an agent still running at the cap fails as “never finished”.
3. Each cell’s clock moves one simulated second per wall second. -pace 0 keeps it manual, and -clock-step then moves it that far each wall second. -seed pins every cell’s random stream (default 1).
4. A cell passes on its verdict, never on the agent’s exit code. Any cell that is not PASS exits 7.
5. -ui serves a grid page on a local port: rows go green or red as cells finish. Run selected and Run failing rerun those cells with the agent folder as it is now, and Run all reruns every cell. It serves until you interrupt it.

Each cell keeps its files under <out>/<eval>/: cell.json, the eval.yaml it ran, agent.log, verdict.json and per simulator twins/<name>/before.json, after.json, calls.json, egress.json, world.json and twin.log. <out>/ledger.jsonl holds one row per run of a cell, with its verdict, reason and checks digest.

Grade a saved run

rystic grade takes cell directories or a grid’s -out directory, and grades each cell from its saved files, never a live simulator:

rystic grade run1
# -> PASS  new_customer
# -> 1 of 1 cells passed
jq -c '{verdict, reason}' run1/new_customer/verdict.json
# -> {"verdict":"PASS","reason":""}

It rewrites each cell’s verdict.json, exits 7 when a cell is not PASS and 1 when one cannot be graded. ERROR means the infrastructure failed, not the agent.

Your own checks: checks/

Checks your evals all share go directly in the agent folder’s checks/: executable scripts and YAML files holding a checks list and an optional allow_delete. Each one runs in every cell; a subfolder runs only where a check names it. Grid copies the folder into each cell, and they run after the eval’s own checks. A script a YAML file names is relative to checks/:

mkdir -p add-customer/checks
printf '%s\n' 'checks:' \
  '  - forbid: {ref: stripe.after.state_customer, new: true, where: {email: {contains: example.com}}}' \
  '    desc: created a customer with a placeholder email' > add-customer/checks/real-emails.yaml
rystic grid add-customer -evals evals -out run2; echo "exit $?"
# -> grid: add-customer × 1 evals, 2 at once → run2
# -> ...FAIL  new_customer ... created a customer with a placeholder email...
# -> 0 of 1 cells passed → run2
# -> each cell's verdict.json, agent.log and twins/*/twin.log are under run2
# -> exit 7

Draft an eval from a good run

rystic grade --draft <cell> grades nothing. It prints an eval drawn from one finished cell, with the cell’s scenarios, prompt and injected events:

rystic grade --draft run1/new_customer > evals/new_customer_draft.yaml
head -3 evals/new_customer_draft.yaml
# -> # DRAFT: rystic grade --draft drew these checks from one recorded run.
# -> # Agent: add-customer. Eval: new_customer.
# -> # They expect every row the run created or changed, forbid every other

Its checks expect each row the run created or changed, and forbid any other new row, changed field, caller or method. allow_delete names each row the run deleted or changed destructively. Ids and times the run generated stay out. A run that never finished, or that would fail its own draft, prints no draft and exits with an error. Trim the checks to what the task requires before you keep the file.

Command evaluators

A simulator with an evaluation engine, today GitHub for its workflow runs, hands the commands it needs run to rystic evaluator. It claims each one, runs it in a digest-pinned container with no network and reports the outcome:

rystic pull github-pulls-twin
rystic run github-pulls-twin -d --name ci --port 0
rystic evaluator --instance ci --once 2>&1
# -> ...evaluator: completed 0
rystic stop ci

1. It runs until Ctrl-C; --once runs what is waiting, then exits. --url <base> names a simulator that rystic ps does not list.
2. A command exits 0 to pass, 1 to fail and 78 to decline the input, or writes its outcome to the file $EVAL_RESULT names. Anything else is an infrastructure error.
3. --runner is docker (the default) or podman. --runner local --trusted-local runs local commands with host access, for trusted code only. --concurrency sets how many run at once (default 2).