CI results and the evaluation engine

The simulator never runs a workflow job: its evaluation engine decides each job’s result. Push a workflow and read what the engine decided for its job:

H='Authorization: Bearer ghp_anything'
R="$RYSTIC_URL/repos/tb-owner/tb-repo"
sha=$(curl -s "$R/branches/tb-base-branch" -H "$H" | jq -r .commit.sha)
curl -s -X POST "$R/git/refs" -H "$H" -d "{\"ref\":\"refs/heads/ci-demo\",\"sha\":\"$sha\"}" | jq -c '{ref}'
# -> {"ref":"refs/heads/ci-demo"}
wf=$(printf 'name: ci\non: [push]\njobs:\n  test:\n    runs-on: ubuntu-latest\n    steps:\n      - run: go test ./...\n' | base64 | tr -d '\n')
curl -s -X PUT "$R/contents/.github/workflows/ci.yml" -H "$H" -d "{\"message\":\"add ci\",\"content\":\"$wf\",\"branch\":\"ci-demo\"}" | jq -c '{commit: .commit.sha[0:8]}'
# -> {"commit":"e24228f3"}
curl -s "$R/commits/ci-demo/check-runs" -H "$H" | jq -c '[.check_runs[] | {name, conclusion, title: .output.title}]'
# -> [{"name":"test","conclusion":"neutral","title":"Not evaluated in this simulation"}]
curl -s "$RYSTIC_URL/_rystic/eval/results" | jq -c '[.results[] | {job: .request.job.id, reason: .result.reason}]'
# -> [{"job":".github/workflows/ci.yml#test","reason":"no_evaluator"}]
curl -s "$RYSTIC_URL/_rystic/eval/policy" | jq -c '.policy.ladder'
# -> ["cache","surrogate","skip"]

The job id is what an evaluator’s supports.job_ids names, and no_evaluator is why its check run reads neutral. The policy’s ladder is the order the engine tries its sources in. How workflows start, and how runs re-run and cancel, is on Workflows and CI runs.

How a result is decided

Each run’s result comes from the evaluation engine (GET /_rystic/eval), which tries these sources in order:

1. a verdict recorded for the same job on the same tree, where only harness-seeded check history and explicit /_rystic/eval/reference imports count;
2. a surrogate evaluator you register;
3. nothing.

Creating or editing an ordinary check run never adds trusted history. How a result was reached never appears on the run.

A surrogate is a command you register and the simulator dispatches. It may be a cheap stand-in rather than the full check. Docker is the default runner, and rystic evaluator --url $RYSTIC_URL runs the command on the job’s snapshot. Replace the image placeholder below with your evaluator image’s immutable reference, including its toolchain and dependencies:

curl -s -X POST $RYSTIC_URL/_rystic/eval/evaluators -d '{"schema_version":2,
  "id":"go-test","version":"1","supports":{"job_ids":[".github/workflows/ci.yml#test"]},
  "requires":{},"implementation":{"kind":"command","command":["sh","-c","go test ./..."],
    "image":"YOUR_IMAGE@sha256:YOUR_64_HEX_DIGEST","platform":"linux/amd64"}}'
curl -s -X POST $RYSTIC_URL/_rystic/eval/policy -d "$(curl -s $RYSTIC_URL/_rystic/eval/policy |
  jq '.policy | .surrogates = [{"id":"go-test","version":"1"}]')"
rystic evaluator --url $RYSTIC_URL &   # exit 0 pass, 1 fail, 78 decline

The snapshot keeps executable file modes and symlinks. It leaves out a symlink whose target leaves the checkout, and leaves a submodule’s directory empty, as actions/checkout does without submodules. A job’s log is the evaluator’s output, the log field of its evidence.

Job ids

A job’s id is <workflow path>#<job id>. A matrix combination adds its values as JSON pairs in brackets, as in .github/workflows/ci.yml#test[[["os","ubuntu-latest"]]]. Copy the exact request.job.id from GET /_rystic/eval/results into supports.job_ids or a task requirement.

Evaluation episodes

An RL task selects task.scope as <owner>/<repo>:<ref>, such as tb-owner/tb-repo:refs/heads/feature. For tb-owner/tb-repo:refs/pull/7/merge, the snapshot is the pull request’s merged tree. Each push or pull request update records its current snapshot even if no workflows match. Removed workflows and deleted refs therefore cannot borrow a previous success.

An RL harness works in three steps:

• it freezes an episode with POST /_rystic/eval/episode;
• it publishes only when it advances, with POST /_rystic/eval/advance;
• it rewards on GET /_rystic/eval/evidence.

A re-run’s results count toward the evidence as the first attempt’s do. rystic help evaluator covers the worker.