Network profiles and fault injection

A network profile adds latency and refuses a fraction of requests, per endpoint, inside the simulator’s own HTTP path. Shaping is off by default and costs nothing until you turn it on. It needs the faults feature, which Pro and Enterprise include.

Only Kalshi ships a profile, off by default (Kalshi → Production latency). RYSTIC_NETWORK=on or POST /_rystic/network with {"enabled":true} turns it on. On any other simulator, {"enabled":true} alone answers 400 cannot enable shaping: no network profile installed, so post a profile.

The profile

POST /_rystic/network
{
  "enabled": true,                     // optional; at least one of enabled / profile
  "profile": {                         // optional; replaces the active profile whole
    "id": "any-name",
    "default": SHAPE,                  // optional; any endpoint not listed below
    "endpoints": {
      "POST /v1/charges": SHAPE        // "METHOD template", exactly as GET /_rystic/routes lists it
    }
  }
}

SHAPE  = { "in": DELAY, "out": DELAY, "reject": REJECT }       // each optional
DELAY  = { "base_ms": 120, "jitter_ms": 40, "correlation": 0.3 }
       | { "quantiles_ms": { "p50": 90, "p90": 180, "p99": 400 }, "correlation": 0.3 }
REJECT = { "rate": 0.25, "status": 429, "body": { … } }        // or "body_text": "…"; "content_type" overrides

1. Keys are METHOD /template exactly as GET /_rystic/routes returns them: POST /v1/charges, GET /v1/customers/{customer}. An endpoint without a key uses default, or nothing.
2. in delays before the handler runs. out delays after state has changed, before the client sees the answer.
3. A delay is base_ms ± jitter_ms, or a quantiles_ms table (p50 required, p90 and p99 optional). correlation in [0,1) makes consecutive delays drift together instead of jumping independently.
4. reject answers rate (0 to 1) of requests with status and exactly the body (JSON) or body_text you give. The type is application/json or text/plain unless content_type says otherwise.
5. A rejected request never reaches the handler, so no state moves. Delays still apply. It sets no other headers, so there is no Retry-After.
6. Deterministic. Delays and rejections draw from their own seeded stream. The same seed-rng and the same profile refuse the same requests in the same order on every run, and shaping never shifts the ids or values the vendor routes draw.
7. Lifetime. A posted profile survives reset, and {"enabled":false} switches it off and keeps it. A scenario that carries a profile, such as Kalshi’s rate_limited, replaces it when it loads or at the times its tape names. The next reset or scenario brings back the profile from before the scenario. A network write made since, a post to /_rystic/network or a posted scenario file with a network, stays instead.
8. Scope. Network profiles never shape the control plane, the WebSocket feed or the replay routes.

GET /_rystic/network returns {enabled, source, profile}.

Worked example: throttle Stripe charges with 429s

Start Stripe on a free port and keep the port in PORT (pull first, so run prints nothing but the JSON):

rystic pull stripe-twin
PORT=$(rystic run stripe-twin -d --name faults --port 0 --scenario design_studio --json | jq -r .data.port)

Refuse half of POST /v1/charges with a Stripe-shaped rate-limit error, and add 120 ± 40 ms on the way back:

curl -s -X POST "localhost:$PORT/_rystic/network" -H 'content-type: application/json' -d '{"enabled":true,"profile":{"id":"charges-throttled","endpoints":{"POST /v1/charges":{"reject":{"rate":0.5,"status":429,"body":{"error":{"type":"invalid_request_error","code":"rate_limit","message":"Request rate limit exceeded."}}},"out":{"base_ms":120,"jitter_ms":40}}}}}'
curl -s "localhost:$PORT/_rystic/network" | jq -c '{enabled, source}'
# -> {"enabled":true,"source":"control"}

Send ten charges and count the answers by status:

for i in $(seq 10); do curl -s -o /dev/null -w '%{http_code}\n' "localhost:$PORT/v1/charges" -u sk_test_twin: -d amount=2000 -d currency=usd -d source=tok_visa; done | sort | uniq -c | awk '{print $2, $1}'
# -> 200 6
# -> 429 4

A rate of 0.5 is a draw per request, not every other one, and the same six pass on every run. Only the 200s created charges. Point your client at localhost:$PORT and check that it backs off and retries.

Then switch shaping off, since reset keeps a posted profile, and wipe the charges:

curl -s -X POST "localhost:$PORT/_rystic/network" -d '{"enabled":false}'
curl -s -X POST "localhost:$PORT/_rystic/reset"
curl -s "localhost:$PORT/_rystic/network" | jq -c '{enabled, source}'
# -> {"enabled":false,"source":"control"}
rystic stop faults

A 503 brownout is the same shape with an HTML page: "reject": {"rate": 1, "status": 503, "body_text": "<html>…</html>", "content_type": "text/html"}.

In the state UI

The state UI’s Network card edits the installed profile while the simulator runs. Install one to edit, here a quantile table for Stripe’s customer list, then open http://localhost:$PORT/_rystic/ui in a browser:

PORT=$(rystic run stripe-twin -d --name netui --port 0 --json | jq -r .data.port)
curl -s -X POST "localhost:$PORT/_rystic/network" -d '{"profile":{"id":"slow-customers","endpoints":{"GET /v1/customers":{"out":{"quantiles_ms":{"p50":100,"p90":250,"p99":600}}}}}}'
curl -s "localhost:$PORT/_rystic/network" | jq -c '{enabled, source, id: .profile.id}'
# -> {"enabled":false,"source":"control","id":"slow-customers"}

1. shaping ON / shaping OFF switches shaping, as {"enabled": …} does. With no profile installed, the button is disabled.
2. Each endpoint row, and a default row when the profile has one, shows its in and out delay in milliseconds. Type a new value and press Enter, and the card posts the whole profile back.
3. An edit scales that delay’s distribution: typing 200 here makes p90 500 and p99 1200, and a base_ms edit scales jitter_ms with it. A delay at 0 is set flat to the new value.
4. edited · revert appears once the profile differs from the one the card read when the page loaded, and posts that one back.
5. An edit takes effect only while shaping is on. The card edits delays, not reject.

Done with the card: rystic stop netui.

At boot

rystic run <product> --network <value>, or RYSTIC_NETWORK=<value> for a container or the binary:

A profile with a default but no endpoints key refuses the boot with network body must set enabled and/or profile. Add "endpoints":{}, or wrap it as {"enabled":true,"profile":{…}}.

On a plan without the faults feature, any value refuses the boot: Boot configuration.

Other faults

The faults feature also revokes credentials mid-run (/_rystic/revoke), duplicates webhook deliveries (/_rystic/duplicate-delivery) and caps page sizes (/_rystic/page-cap). Control plane → Routes by plan has each one’s body.