The hard part of testing is no longer finding bugs

AI has made finding and fixing bugs cheap and efficient. The challenge has moved on to efficiently figuring out what behavior is actually a bug.

Deciding that means generating intents: statements of what the product should do.

The usual definition of a bug is behavior that deviates from the spec. But users never read the spec, so from their perspective a bug is any behavior that doesn’t do what they expected.

Bugs then rest on what the owner intended. Some intents are clear, forms should record their results, apps shouldn’t crash on launch, and dropdowns should be easy to navigate. Others aren’t: should a subscription canceled halfway through the month get a prorated refund? Should a country dropdown put the US first?

• User submits a form → the result is properly recorded
• User opens a dropdown → options are in a sensible order
• User searches with a typo → search still finds the result

Every intent has two halves: what the user does and what should happen when they do it. Generating the first half has become cheap, moving the bottleneck to the judgement of what should happen.

If we want to produce high-quality software, we need a system for generating intents that is:

• Proactive: bugs should be caught before release.
• Easy: it fits the software you already have without major modification.
• Realistic: its scenarios match what the system will have to handle in the real world.

These then should be dealt with by a system that is able to make reasonable judgements of non-core product decisions.

If the system cannot do this then the human will be stuck making these decisions when they do not have to.

Real user intents

The obvious source of intents is your users. They are a representative sample of the real world. You can spot bugs more easily than before: users report them, and rage clicks flag them. AI has made instrumentation cheap, so recreating a user’s state is simple.

That makes a loop easy to build:

1. Take intents in (via Sentry and PostHog)
2. Reproduce the bug (in a microVM)
3. Fix it (with an agent)
4. Verify the fix (in a microVM)
5. Merge (after automated review).

Every team with outwards facing software should be running this loop.

The catch is that real intents only arrive after release. You can’t use them to test software before it ships, or to test a new feature. And you need users willing to put up with buggy new software.

Verdict: users supply both halves, since they complain when the result is wrong, and their scenarios are a representative sample. But they only arrive after release.

Fake user intents

If real users intents arrive too late one solution is to fake them via simulated users. Good simulation will give you breadth and depth, and follow typical user paths.

But it finds a narrow range of bugs. If your product is a database or accounting software rather than a UI people click through, simulated users have nothing to do. Additionally they cannot tell what the user expects if the system is complex, for example with refunding a subscription canceled halfway through the month.

This approach will work for self-contained, UI-driven products and little else.

Verdict: proactive and easy to set up for UI products, but it supplies only the action half. It catches a crash, not that the dropdown could be ordered better.

All the intents

Another way of generating intents is to generate all the possible intents against an expected output. This is what companies like Worksers io do by running your entire system in a deterministic simulation. They put your code in a hypervisor then stress it with fault and bad inputs, testing every possible path.

Trying everything is very proactive. But it is hard to set up and hard to extend. For anything meaningful you have to simulate its outside dependencies, and for ordinary software that isn’t worth it, even with AI writing the code. And you still define the expected output of every behavior, so it can’t tell you when something is off in qualitative ways.

Verdict: this approach supplies every action, but you hand-write every expected result, which is why it is the hardest to set up. Its scenarios are exhaustive rather than realistic.

The ideal solution

The ideal solution does not exist yet.

Each approach covers some of the needs and misses others. The ideal solution would generate both halves: realistic actions before release, and expected results it fills in on its own for minor calls while presumably sending the essential ones to you.

No one has built this yet, at least not in a repeatable way.

We are taking steps towards this @rystic by creating deterministic simulations of all the third party services like Stripe that a system depends on, but getting to an ideal solutions will still take time and iteration.

Until then, here is how to test your own system:

1. Write smoke tests and user journeys, using comparable products to estimate volume. This will give you realistic data before release.
2. For minor calls, take the result half from what other people do. There are many great products out there, do what they do instead of trying to innovate on how you refund a subscription canceled halfway through the month.
3. Decide the result half of the essential calls yourself. Don’t let AI make them. It is your product and you need to own the key decisions.