rystic

Blog

Rystic vs WireMock for stateful API testing

By Rystic ·

For integration tests and agent evals against a stateful API like Stripe, use a simulator that enforces the API’s rules, not a recording. In our tests, WireMock’s recorded stubs were right only for the exact calls they recorded. WireMock Cloud’s Stripe template kept state, but accepted an over-refund, a repeat refund and a reused idempotency key. The Rystic Stripe simulator returned the error Stripe documents in each of those cases. WireMock is still a good self-serve option for what Rystic doesn’t cover.

The difference is who writes the behavior. WireMock keeps state: its scenarios are “essentially a state machine” [1], an official extension carries data between stubs [2], and WireMock Cloud adds Dynamic State and a stateful Stripe template [6][10]. You, or a template author, write the stubs and the state transitions. Rystic ships a pre-built simulator of each API it covers, with realistic data already loaded and fault injection built in, and each simulator is checked against the live API before release.

On 10 November 2026 we ran the Rystic Stripe simulator, WireMock OSS and WireMock Cloud’s Stripe template on the same 9-step payments flow, then on eight requests that step off it.

The short version:

  • The payments flow, scored against Stripe’s recorded answers. The Rystic simulator matched on 8 of 9 steps. WireMock OSS, replaying a recording, matched on 8 of 9 on its first run and 6 of 9 on a second run without a reset. WireMock Cloud’s Stripe template matched on 3 of 9. Two of the template’s six misses are limits its own page states, and one depends on which declined test card you send.
  • Eight off-script requests. Recorded WireMock stubs answered six with WireMock’s own “no match” 404. Loosened stubs answered with wrong data and a 200. WireMock Cloud’s template kept real state on five, but accepted an over-refund, a second full refund and a reused idempotency key. The Rystic simulator gave a Stripe-style answer to all eight.
  • Where WireMock wins. It is a good self-serve option for things Rystic doesn’t cover, like Stripe endpoints outside the simulator’s slice of the API. More below.

How each tool keeps state

WireMock OSS scenarios. “WireMock supports state via the notion of scenarios. A scenario is essentially a state machine whose states can be arbitrarily assigned.” [1] A scenario holds one named state; stubs match on it and move it on. Its limit, in the words of WireMock’s own state extension: “Scenarios cannot transport any data other than the state value itself” [2]. Scenario state is global, so “one user/thread can affect outcomes for another”, as an open proposal from a WireMock maintainer puts it [3]. An issue reporting a race condition on parallel requests has been open since 2016 [4]. A fix making scenario updates atomic was merged on 27 October 2026 and shipped in 4.0.0-beta.42 on 12 November [5]. It is not in 3.13.2, the latest stable release and the version we tested.

Record and playback. WireMock can record traffic to a real service and turn repeated identical requests into a scenario, so “a series of requests matching this stub will yield the same series of responses captured during recording” and “the last response will continue to be returned” after that [7].

wiremock-state-extension. In the official wiremock GitHub org, it adds keyed contexts, recorded and deleted state, a state matcher and a template helper [2]. Its README is plain about the limits: it is “not optimized for distributed setups or high degrees concurrency”, “the context can change while a request is performed”, “The default expiration is 60 minutes”, and “List elements cannot be modified (only read/deleted).”

WireMock Cloud Dynamic State. WireMock describes it as “a more sophisticated and powerful replacement for the existing Scenarios functionality”: string values under a key, within a context, per mock API [6]. Values live in “a Least Recently Used cache, and no guarantee is made about how long they will persist.” Enterprise keeps up to 100,000 values, and other plans with Dynamic State enabled keep 100 [8]. Its stateful endpoint generator creates seven CRUD stubs, and the docs add: “You will want to further modify the stubs to better simulate the API’s real-world stateful behavior.” [9] On the published pricing, the Free plan does not include stateful mocking; Enterprise does, along with chaos testing [11].

WireMock Cloud’s Stripe template. The strongest stateful option WireMock offers for Stripe. It “covers every operation in Stripe’s published OpenAPI description for API version 2026-09-30.endive”, 1,062 stubs across 644 endpoints, “and every resource it creates is stored” [10]. Before publication, “an AI agent exercised these endpoints against the real Stripe API through WireMock’s recording proxy”, and the recorded traffic verified each stub’s request and response shapes. The page states its own limits: “No events are delivered”, an Idempotency-Key header “is accepted without deduplication”, pagination always reports has_more false, and the public sandbox is “one shared store”. For state you can reset, it says to “open the template in your own WireMock Cloud account”. It also says declines are modeled: “the pm_card_chargeDeclined family returns a 402 card_error with the matching decline_code”.

The Rystic Stripe simulator. A local server that keeps state and runs Stripe’s rules for the 196 operations it serves, 181 of Stripe’s 612 v1 operations and 15 of 32 v2 [13]. Nothing is hand-authored per test: it boots with the design_studio scenario, a studio’s Stripe account with 30 customers and 269 charges already in it (Stripe simulator docs). A control plane resets it to the same ids, seeds state, moves the clock and injects faults: 429s, latency, revoked keys and duplicate webhooks. The same starting state and the same requests give byte-identical ids, timestamps and events, run after run (Reset and determinism). We check it against live Stripe daily. When we published our Stripe deep dive on 25 October 2026, 99.1% of 1,813 daily checks matched on the operations it serves [13].

Test setup

  • Rystic: stripe-twin 0.0.25, run natively with rystic run.
  • WireMock OSS: 3.13.2, from the official Docker image wiremock/wiremock:latest, with default flags. It recorded the flow once, then replayed it. The recording produced 14 stub mappings in 375 lines of JSON, with 3 scenarios generated automatically.
  • WireMock Cloud: the public sandbox of WireMock’s own Stripe template, https://library-jy0k6.wiremockapi.cloud, shared and open without an account [10].
  • Flow: the 9-step payments flow from our public benchmark repo, stripe/harness/run_payments.py, unmodified [16]:
    1. Create a customer.
    2. Pay $50, then send the same request again with the same idempotency key.
    3. Read the charge and its balance transaction.
    4. Refund $20.
    5. Refund the rest.
    6. Try to refund again.
    7. Pay with a declined card.
    8. List the customer’s charges.
    9. List the events.
  • Referee: Stripe’s answers to the same flow, recorded in Stripe test mode on 20 October 2026 and kept in the same repo as stripe/results/2026-10-20/runs/payments-stripe.json. The repo’s compare_payments.py scores each step field by field [16].
  • API version: every call sends Stripe-Version: 2026-06-24.dahlia, the version Stripe’s answers were recorded on.

Test 1: the payments flow

Each column is scored against Stripe’s recorded answers. “No reset” means the flow ran a second time against the same server, without returning it to its starting state.

StepRystic 0.0.25Rystic, 2nd run, no resetWireMock OSS replay, 1st runWireMock OSS replay, 2nd run, no resetWireMock Cloud Stripe template
1. Create a customermatchmatchmatchmatchmatch
2. Pay $50, then retry with the same keymatchmatchmatchmatchdiffers: the retry made a second payment
3. Read the charge and its balance transactionmatchmatchmatchmatchdiffers: balance transaction amount, fee, net, type
4. Refund $20matchmatchmatchdiffers: the charge’s refunded amountsmatch
5. Refund the restmatchmatchmatchdiffers: refund amount, statusmatch
6. Try to refund againmatchmatchmatchmatchdiffers: the refund succeeded, where Stripe returns an error
7. Pay with a declined cardmatchmatchmatchmatchdiffers: the card was charged
8. List the customer’s chargesmatchmatchmatchmatchdiffers: three succeeded charges, where Stripe lists one failed and one succeeded
9. List the eventsdiffers: charge.updated missingsamesamesamediffers: none of the flow’s events listed
Agrees with Stripe8/98/98/96/93/9

The recording replays exactly, as WireMock’s docs say it will, but only for the exact recorded sequence starting from a reset. On the second run without a reset, the scenarios had already reached their last recorded response, so the refunds came back wrong.

WireMock OSS’s 8 of 9 is the Rystic simulator’s 8 of 9: it recorded the simulator, so it inherits the simulator’s answers, including its one miss, the charge.updated event (caveat 1).

The Cloud template missed six steps, for four different reasons:

  • Steps 2 and 9 are limits its page states: no idempotency deduplication and no events [10].
  • Step 8 follows from steps 2 and 7. The list carries the second payment from step 2 and the charge that should have failed in step 7.
  • Step 7 depends on the test card. Our flow sends pm_card_visa_chargeDeclined, the PaymentMethod Stripe’s testing docs list for a generic decline [19]. Stripe declined it in the recorded run, and the template charged it. The template does decline pm_card_chargeDeclined, as its page says: when we sent that id on 16 November 2026 it returned Stripe’s 402 card_declined.
  • Steps 3 and 6 are not on its list of limits. The balance transaction it returned was not the charge’s, and the third refund succeeded.

Test 2: eight requests off the recorded path

A test suite that changes one amount, or an agent that takes a different path, steps off the recording. Each request below starts from a reset: Rystic’s POST /_rystic/reset, WireMock’s POST /__admin/scenarios/reset. The public Cloud sandbox can’t be reset, so each request there worked on objects it had just created.

“Loosened” stubs are the recorded stubs with body matchers removed and ids turned into patterns, standing in for what a user does to make recorded stubs reusable (caveat 4).

RequestRystic 0.0.25WireMock OSS, as recordedWireMock OSS, loosenedWireMock Cloud Stripe template
A. Create a customer with a different email, read it back200, ada@example.com both times404, not Stripe-shaped200 but returns jenny.rosen@example.com; read-back 404200, correct email both times
B. Pay $75 instead of $50200, amount 7500, succeeded404, not Stripe-shaped200 but amount 5000200, amount 7500
C. Refund $15, read the chargerefund 1500; charge amount_refunded 1500404, not Stripe-shapedrefund 2000; charge shows 0refund 1500; charge shows 1500
D. Refund $60 of a $50 charge400 invalid_request_error; charge unchanged404, not Stripe-shaped200, no error200, no error; charge amount_refunded 6000
E. Two payments, refund the second, read bothdistinct ids; 2nd refunded, 1st notsame id twice; refund 3000; neither refundedsame as recordeddistinct ids; 2nd refunded, 1st not
F. Reuse an idempotency key with a different amount400 idempotency_error404, not Stripe-shaped200, amount 5000200, new payment for 9900
G. Read a charge that was never created404 resource_missing404, not Stripe-shaped200404 resource_missing
H. Refund the same payment twice in full2nd refund 400 charge_already_refunded1st refund returns 3000, not 5000; 2nd refund 400 charge_already_refunded; charge shows 0same as recorded2nd refund 200, no error

There is no live-Stripe run of these requests (caveat 2), so “right” here means what Stripe documents:

  • Refunds (D, H). Creating a refund “will raise an error when called on an already-refunded charge, or when trying to refund more money than is left on a charge” [14].
  • Idempotency keys (F). “The idempotency layer compares incoming parameters to those of the original request and errors if they’re not the same” [12].
  • Unknown ids (G). They get resource_missing [15].

We ran the Cloud column again on 16 November 2026 and got the same eight answers.

What the eight requests show:

  • Recorded stubs answer only the requests they recorded. Six of eight got WireMock’s “no match” 404, and the other two (E and H) got answers recorded for a different request.
  • Loosened stubs stop returning 404 and start returning wrong answers with a 200, including for an over-refund and a charge that doesn’t exist. A test that checks only the status code passes against them.
  • WireMock Cloud’s Stripe template keeps real per-resource state (A, B, C, E and G are right) but didn’t enforce three of Stripe’s rules: it accepted an over-refund, a second full refund and a reused idempotency key. Those are the cases a double-refund or double-charge test exists to catch.
  • The Rystic simulator gave a Stripe-style answer to all eight, errors included.

Caveats

  1. No Stripe test key was available, so WireMock OSS recorded the Rystic simulator, not real Stripe. Its replay inherits the simulator’s answers, including the simulator’s one miss on step 9.
  2. The eight off-script requests have no live-Stripe referee. They show what each tool returned; “expected” is Stripe’s documented behavior, not a recorded Stripe run.
  3. The WireMock Cloud flow result is one run against a shared public sandbox on one day. Other visitors’ objects share its store [10]. We repeated only the eight off-script requests, on 16 November. A private copy of the template in your own account may behave differently. We didn’t test one, and we didn’t confirm which plan its state needs.
  4. The loosened stubs are our edit: body matchers removed and ids turned into patterns, standing in for what a user does to make recorded stubs reusable. Loosening them another way would give other answers.
  5. We didn’t test a hand-authored WireMock OSS build with the state extension, or the Cloud Stripe template in a private account. With enough stubs, either could enforce the rules these requests check; the question is who writes and maintains them.
  6. The declined-card step depends on the card id. Our flow sends pm_card_visa_chargeDeclined, which the template charged. It declines pm_card_chargeDeclined with Stripe’s 402 card_declined. We didn’t rescore the flow with that id, and we expect step 8 to differ either way, because of the second payment from step 2.
  7. The API versions differ. Our calls send Stripe-Version: 2026-06-24.dahlia, and Stripe’s answers were recorded on that version. WireMock’s template is built from Stripe’s 2026-09-30.endive description [10]. We didn’t check whether any difference in the table comes from the version.

The payments harness and Stripe’s recorded answers are public in Rystic-Inc/rystic-benchmarks  [16]. The script for the eight off-script requests, the WireMock stubs and the run files from this comparison are not in that repo yet.

What this means for integration tests and agent evals

A recorded stub set is a replay of one conversation. It catches regressions on that one path. Change an amount, add a refund or run twice without a reset, and it answers with the wrong data or a 404. You can write the state transitions by hand, with scenarios or the state extension, but every rule Stripe enforces, from idempotency to the refund ceiling, is one more stub you write and keep current.

WireMock Cloud’s Stripe template removes most of that writing and keeps real state. In our run it didn’t reject an over-refund, a repeat refund or a reused idempotency key, and its page says plainly that it sends no events and doesn’t deduplicate retries.

An agent eval leans on those rules harder than a test suite does. An agent takes paths nobody recorded, so the API it calls has to answer unscripted requests the way the real one would, errors included, and start every run from the same state. That is what the Rystic simulator is built for: a reset to the same ids, byte-identical runs, and simulators of its own for each eval run, so no two runs share state. For CI, the same simulator runs as a service container.

When WireMock is the better choice

  • An API Rystic doesn’t cover. WireMock mocks any HTTP API, if you have the time to set the mock up, you’re not limited by waiting for a company to implement something for you.
  • Stripe breadth. WireMock’s template claims every operation in Stripe’s spec [10]; the Rystic simulator serves about a third of Stripe’s API. Issuing, Terminal and Treasury are among the Stripe products the simulator doesn’t serve yet [13]; the template covers them [10].
  • Exact replay and request matching. If you need precise request matching, selective proxying to the real service, or one recorded sequence replayed exactly, WireMock does that well. It replayed our recording exactly.
  • Open source and free. WireMock OSS is Apache-2.0, with over 6 million downloads a month by its README’s count [17] and a Java extension model. Rystic is closed source and licensed, and a fully offline machine needs the Enterprise plan.

Try it on your own flow

Our flow is one sample. Point your own suite at the Rystic Stripe simulator and at your WireMock stubs, and see which failures are real. If we ran a tool wrong or misread a result, open an issue in Rystic-Inc/rystic-benchmarks  or email founders@rystic.ai.

Sources

  1. WireMock docs: stateful scenarios : “A scenario is essentially a state machine whose states can be arbitrarily assigned.”
  2. wiremock-state-extension README : what scenarios and templating can’t carry, and the extension’s concurrency, expiry and list limits
  3. wiremock#2684 : an open proposal to scope scenario state, which “applies globally to the mock API” today
  4. wiremock#456 : “parallel request in scenarios (stateful) - race condition”, open since July 2016
  5. wiremock#3641 : atomic scenario updates, merged 27 October 2026 and listed in the 4.0.0-beta.42 release notes 
  6. WireMock Cloud: Dynamic State 
  7. WireMock docs: record and playback 
  8. WireMock Cloud: Dynamic State plan limits 
  9. WireMock Cloud: create a stateful endpoint set 
  10. WireMock Cloud Stripe template and live sandbox : coverage, verification method, what it models and its stated limits
  11. WireMock Cloud pricing 
  12. Stripe API: idempotent requests 
  13. Deep dive: Stripe simulator vs mock vs sandbox: the simulator’s coverage, what it doesn’t serve yet, and its daily fidelity checks
  14. Stripe API: create a refund : errors on an already-refunded charge and on refunding more than is left
  15. Stripe error codes : resource_missing, charge_already_refunded
  16. Rystic-Inc/rystic-benchmarks : the payments harness, compare_payments.py and Stripe’s recorded answers
  17. wiremock/wiremock README 
  18. WireMock docs: simulating faults 
  19. Stripe docs: testing : pm_card_visa_chargeDeclined is the PaymentMethod listed for a generic decline

Stripe is a trademark of Stripe, Inc. Rystic is not affiliated with or endorsed by Stripe or WireMock.