26 Jun 2025 AI Engineering 5 min read

Service Virtualization for Agents

Agent journeys end at systems you cannot test against at volume: payment providers, ERP backends, partner APIs. We apply service virtualization to agents — simulated counterparts implementing the real contract, holding state, injecting faults — and argue that every simulated result must carry an explicit label. With verified data from τ-bench, τ²-bench, and Microcks 1.12.0.

Testing against systems you cannot touch

An agent journey is the full path an agent takes through a task: model calls, tool calls, retries, and the state changes they cause. Many of the tools at the end of that path belong to someone else. Payment providers, ERP backends, carrier APIs, government registries. You cannot run five hundred evaluation episodes against them. Rate limits, contractual terms, irreversible side effects, and per-call cost forbid it. Yet the journey is only tested if those calls happen.

This is not a new problem. Classic integration testing hit the same wall two decades ago and answered it with service virtualization. What is new is the client. An agent is a non-deterministic client, and that changes what a useful simulation looks like. This article describes the adaptation, and what it does not buy you.

A task arrives — the agent plans its next step. 1/4

Service virtualization in brief

Service virtualization means replacing a dependency with a simulated counterpart that speaks the dependency's real protocol and honors its real interface. The consumer under test is not modified and does not know it is talking to a simulation. Tools such as WireMock, Mountebank, Hoverfly, and Microcks have carried this pattern for years: record or specify behavior, serve it behind the real endpoint shape, assert against what the consumer did.

The distinction from ad-hoc mocking matters. A mock lives inside your test code and mirrors your assumptions. A virtualized service lives behind the wire and mirrors the contract. Only the second can be shared between teams, versioned independently, and pointed at by an unmodified agent running its real tool stack.

Why agents break classic mocks

Classic stubs assume a deterministic client: known call order, known payloads, one path. Agents violate every one of these assumptions. They choose call order at runtime, paraphrase parameters, retry after failures, and explore. τ-bench demonstrated this in June 2024: state-of-the-art function-calling agents solved fewer than 50% of its tasks, and repeated runs of the same task diverged — pass^8 fell below 25% in the retail domain.

A stub that answers one expected request therefore covers almost nothing. The simulated counterpart must behave like a service: accept any contract-valid request, maintain state across calls, and reject invalid requests the way the real system would. Otherwise the agent passes tests by walking a corridor that does not exist in production.

Property	Deterministic client	Agent client
Call order	Fixed and known upfront	Chosen at runtime
Payloads	Exact expected values	Any contract-valid variant
Retries	Scripted	Emergent and model-dependent
Coverage target	One request-response pair	The behavior space of the contract
Test oracle	Response equality	End state and journey checks

Simulated counterparts behind real contracts

The stable element in this setup is the contract: an OpenAPI 3.1 document, a gRPC proto, an AsyncAPI channel, or an MCP tool schema. The simulation implements the contract; the agent consumes the contract; neither knows the other. Every simulated response is validated against the schema, and every agent request is validated the same way. A contract violation fails the test regardless of task outcome, because it predicts a production failure no eval score can excuse.

The tooling caught up this spring. Microcks 1.12.0, released on 28 May 2025, exposes existing OpenAPI, GraphQL, and gRPC mocks as MCP endpoints — over SSE per the 2024-11-05 protocol revision and Streamable HTTP per 2025-03-26. MCP itself was introduced by Anthropic in November 2024. An agent can now discover and call a virtualized backend through the same protocol it would use in production.

Stateful simulation and fault injection

Two extensions separate agent-grade virtualization from request replay. First, state: an order created in call three must be retrievable in call seven, because agents verify their own work and branch on what they find. Second, faults: timeouts, 429 responses, partial failures, and stale reads, injected deterministically — recovery behavior is the part of a journey most worth testing and least visible in happy paths.

The counterpart is sometimes a person. τ²-bench, published on 9 June 2025, simulates users who hold their own tools in a telecom support domain. Agent pass^1 dropped by roughly 20 percentage points once a simulated user entered the loop — GPT-4.1 reached 34% there. Simulated humans belong in the harness for the same reason simulated services do: the real ones are unavailable at test volume.

Labeling simulated results

Every result produced against a simulation must carry that fact as data. We attach a simulation label to each trace, tool response, and evaluation record — an explicit field, not a naming convention. The label propagates: a report aggregated from simulated episodes is itself labeled simulated, and a journey that mixed real and simulated calls is labeled mixed.

The reasons are concrete. First, honest reporting: "the journey passes" silently becomes "the journey passes against our model of the counterpart", and readers of a dashboard must see that difference. Second, contamination: agent memories, few-shot example pools, and fine-tuning datasets built from traces must be able to exclude simulated interactions. An unlabeled simulated success is a small lie with a long half-life.

What virtualization does not prove

Virtualization proves that the agent behaves correctly against your model of the counterpart. It does not prove the model is right. Undocumented behavior, contract drift on the provider side, real latency distributions, and production authentication flows all sit outside the simulation. A contract-valid simulator can still be semantically wrong, and it will be wrong in exactly the places nobody thought to specify.

We therefore treat simulated evaluation as a gate, not a verdict. It runs on every change, cheaply and repeatably. It is followed by a thin layer of supervised runs against real sandboxes where they exist, and by staged rollout with tight monitoring where they do not. Skipping that second layer converts a test strategy into a belief.

The outlook from June 2025

Three developments seem likely from where we stand. First, MCP-native virtualization becomes a category of its own: the protocol has seen three specification revisions in eight months — 2024-11-05, 2025-03-26, and 2025-06-18, published one week ago — and mock tooling is tracking it release by release. Second, API providers will start shipping official simulators the way they ship SDKs today, because agent traffic makes unofficial sandboxes economically untenable.

Third, the simulation label will outgrow the test harness. As agents operate in production with simulated components mixed in — dry runs, shadow tools, rehearsal environments — provenance of results becomes an operational requirement rather than a testing convention. We expect labeling of simulated outcomes to appear in agent observability standards within the next year. Teams that adopt it now will not have to untangle their data later.