# CI results and the evaluation engine

How each workflow job's result is decided without running it (a recorded verdict, a surrogate evaluator you register or neither, which reads not evaluated) and the episodes an RL harness rewards on.

The simulator never runs a workflow job: its evaluation engine decides each job's result. Push a workflow and read what the engine decided for its job:

```bash
H='Authorization: Bearer ghp_anything'
R="$RYSTIC_URL/repos/tb-owner/tb-repo"
sha=$(curl -s "$R/branches/tb-base-branch" -H "$H" | jq -r .commit.sha)
curl -s -X POST "$R/git/refs" -H "$H" -d "{\"ref\":\"refs/heads/ci-demo\",\"sha\":\"$sha\"}" | jq -c '{ref}'
# -> {"ref":"refs/heads/ci-demo"}
wf=$(printf 'name: ci\non: [push]\njobs:\n  test:\n    runs-on: ubuntu-latest\n    steps:\n      - run: go test ./...\n' | base64 | tr -d '\n')
curl -s -X PUT "$R/contents/.github/workflows/ci.yml" -H "$H" -d "{\"message\":\"add ci\",\"content\":\"$wf\",\"branch\":\"ci-demo\"}" | jq -c '{commit: .commit.sha[0:8]}'
# -> {"commit":"e24228f3"}
curl -s "$R/commits/ci-demo/check-runs" -H "$H" | jq -c '[.check_runs[] | {name, conclusion, title: .output.title}]'
# -> [{"name":"test","conclusion":"neutral","title":"Not evaluated in this simulation"}]
curl -s "$RYSTIC_URL/_rystic/eval/results" | jq -c '[.results[] | {job: .request.job.id, reason: .result.reason}]'
# -> [{"job":".github/workflows/ci.yml#test","reason":"no_evaluator"}]
curl -s "$RYSTIC_URL/_rystic/eval/policy" | jq -c '.policy.ladder'
# -> ["cache","surrogate","skip"]
```

The job id is what an evaluator's `supports.job_ids` names, and `no_evaluator` is why its check run reads `neutral`. The policy's `ladder` is the order the engine tries its sources in. How workflows start, and how runs re-run and cancel, is on [Workflows and CI runs](https://www.rystic.ai/docs/simulators/github-pulls/features/workflows.md).

### How a result is decided

Each run's result comes from the evaluation engine (`GET /_rystic/eval`), which tries these sources in order:

1. a verdict recorded for the same job on the same tree, where only harness-seeded check history and explicit `/_rystic/eval/reference` imports count;
2. a surrogate evaluator you register;
3. nothing.

Creating or editing an ordinary check run never adds trusted history. How a result was reached never appears on the run.

| The engine says | The run shows |
|---|---|
| evaluated, pass | `completed` / `success` |
| evaluated, fail | `completed` / `failure` |
| the job's `if:` is false, or a dependency failed or was skipped under the implicit `success()` (a skipped job's outputs read empty) | `completed` / `skipped` |
| canceled | `completed` / `cancelled` |
| the evaluator timed out | `completed` / `timed_out` |
| not evaluated (no evaluator, unsupported job or condition, a dependency not evaluated) | `completed` / `neutral`, "Not evaluated in this simulation" |

A surrogate is a command you register and the simulator dispatches. It may be a cheap stand-in rather than the full check. Docker is the default runner, and `rystic evaluator --url $RYSTIC_URL` runs the command on the job's snapshot. Replace the image placeholder below with your evaluator image's immutable reference, including its toolchain and dependencies:

```sh
curl -s -X POST $RYSTIC_URL/_rystic/eval/evaluators -d '{"schema_version":2,
  "id":"go-test","version":"1","supports":{"job_ids":[".github/workflows/ci.yml#test"]},
  "requires":{},"implementation":{"kind":"command","command":["sh","-c","go test ./..."],
    "image":"YOUR_IMAGE@sha256:YOUR_64_HEX_DIGEST","platform":"linux/amd64"}}'
curl -s -X POST $RYSTIC_URL/_rystic/eval/policy -d "$(curl -s $RYSTIC_URL/_rystic/eval/policy |
  jq '.policy | .surrogates = [{"id":"go-test","version":"1"}]')"
rystic evaluator --url $RYSTIC_URL &   # exit 0 pass, 1 fail, 78 decline
```

The snapshot keeps executable file modes and symlinks. It leaves out a symlink whose target leaves the checkout, and leaves a submodule's directory empty, as `actions/checkout` does without `submodules`. A job's log is the evaluator's output, the `log` field of its evidence.

### Job ids

A job's id is `<workflow path>#<job id>`. A matrix combination adds its values as JSON pairs in brackets, as in `.github/workflows/ci.yml#test[[["os","ubuntu-latest"]]]`. Copy the exact `request.job.id` from `GET /_rystic/eval/results` into `supports.job_ids` or a task requirement.

### Evaluation episodes

An RL task selects `task.scope` as `<owner>/<repo>:<ref>`, such as `tb-owner/tb-repo:refs/heads/feature`. For `tb-owner/tb-repo:refs/pull/7/merge`, the snapshot is the pull request's merged tree. Each push or pull request update records its current snapshot even if no workflows match. Removed workflows and deleted refs therefore cannot borrow a previous success.

An RL harness works in three steps:

- it freezes an episode with `POST /_rystic/eval/episode`;
- it publishes only when it advances, with `POST /_rystic/eval/advance`;
- it rewards on `GET /_rystic/eval/evidence`.

A re-run's results count toward the evidence as the first attempt's do. `rystic help evaluator` covers the worker.

---

Source: https://www.rystic.ai/docs/simulators/github-pulls/features/ci-evaluation · Markdown: https://www.rystic.ai/docs/simulators/github-pulls/features/ci-evaluation.md