07 Jul 2026 AI Engineering 6 min read

Simulation-First Agent Engineering

End-to-end agent journeys terminate in state changes inside authorities, ERPs, and partner APIs — systems you cannot freely test against. This article defines the simulation-first pattern: swap the real counterpart for a stateful simulated one behind the same contract. We position it against service virtualization and digital twins, cover honest labeling, load and adversarial testing, the fidelity gap, and where simulated counterpart ecosystems are heading.

Why Real Counterparts Resist Testing

An agent journey is the end-to-end execution of a business process by a software agent: it collects data, calls external systems, reacts to their responses, and terminates in a verifiable outcome. The counterpart system is the external system the journey runs against — a tax authority's acceptance server, an ERP, a partner API. Because the outcome of a journey is a state change in the counterpart, testing the agent means testing against the counterpart. That is exactly where testing breaks down.

Production counterparts have irreversible side effects: a submitted filing is a legal act, a booked order moves goods and money. Official sandboxes exist, but they are deliberately partial. HMRC's API sandbox omits 2-step verification and identity checks; the Companies House sandbox replaces internal processing with a mock that returns status updates without changing register data. Sandboxes also enforce rate limits, reset test data on their own schedule, and rarely produce failure modes on demand. An agent under development needs hundreds of runs per journey, including deliberately broken ones. No real counterpart — production or sandbox — supports that.

Two peers — each with its own state and private graph. 1/4

The Swap Behind the Contract

Simulation-first agent engineering builds the agent against a contract and makes the counterpart behind that contract swappable. A contract, in this sense, is the machine-readable interface of the counterpart: its endpoints and schemas (typically OpenAPI), its protocol and authentication flow, and — often forgotten — its error semantics. A simulated counterpart is a second implementation of the same contract with its own state store, deployable wherever tests run.

Two rules make the pattern sound. First, the agent must not know which implementation it faces: same tools, same code path, same retry logic; the switch lives in environment configuration, never in the agent's reasoning. An agent that branches on a test mode tests a different agent. Second, the simulator must be stateful. A journey that submits, polls for status, and corrects a rejection needs a counterpart whose answers depend on what happened before. Canned responses test payload parsing, not journeys.

What Service Virtualization Solved

The infrastructure for this is not new. Service virtualization stands up a networked test double that the system under test reaches over the real protocol — distinct from in-process mocking, which patches objects inside the test runner. WireMock provides JSON stub mappings, record-and-playback, stateful scenarios, fault injection, and simulated delays; Hoverfly captures and replays traffic as a Go-based proxy; Mountebank's imposters cover HTTP, TCP, and SMTP. Record-and-replay is the workhorse: capture real traffic once, replay it deterministically forever.

Agents break the workhorse. Record-and-replay assumes the client will repeat the recorded path. An LLM-driven agent does not: it phrases requests differently across runs and chooses different tool sequences toward the same goal. A response archive therefore cannot answer it. The simulated counterpart must be a behavioral model — domain state plus transition rules — that computes responses to requests nobody recorded. Virtualization tools remain useful underneath that model, as the transport and fault-injection layer.

A Digital Twin Is Not a Simulator

The term digital twin does not fit here, and the distinction matters. The Digital Twin Consortium defines a digital twin as "a virtual representation of real-world entities and processes, synchronized at a specified frequency and fidelity" (December 2020); ISO/IEC 30173:2023 standardizes the terminology, and ISO 23247 specifies a reference architecture for manufacturing twins. Synchronization with a real entity is constitutive. A simulated counterpart is deliberately not synchronized: it produces counterfactual state so journeys can run without touching the real system. Calling it a twin claims a fidelity it does not have.

Technique	Replaces	State	Synced with reality	Primary purpose
In-process mock	An object inside the test process	None or minimal	No	Unit-test isolation
Service virtualization	A networked service behind its protocol	Scripted or recorded	No	Integration tests without the dependency
Simulated counterpart	A full counterpart system behind its contract	Behavioral model with evolving state	No — intentionally counterfactual	End-to-end agent journeys
Digital twin	Nothing — it mirrors a real entity	Mirror of real state	Yes at defined frequency and fidelity	Monitoring and prediction

Label Simulated Results as Simulated

A journey produces artifacts: confirmation numbers, transaction IDs, documents, ledger entries. From a good simulator these are indistinguishable from real ones — by design, and that is the hazard. Honest labeling means provenance at the data layer, not the UI layer: every record written during a simulated run carries a machine-readable origin marker, simulated identifiers come from a reserved namespace, and generated documents are visibly watermarked. A badge in the interface does not survive export to a spreadsheet.

Three rules follow. Mark provenance at write time, never at display time. Never aggregate simulated and real outcomes into one metric — a success rate that silently includes simulated confirmations is a false statement. And never let an agent's persistent memory ingest simulated outcomes as facts about the real world; simulated state should be purgeable in a single operation.

Load and Adversarial Testing on Demand

Agent reliability is a statistical property, which makes simulation the only affordable measurement instrument. τ-bench (ICLR 2025) evaluates agents against LLM-simulated users and scores the final database state; its pass^k metric asks whether all k independent trials succeed. GPT-4o with function calling reached about 61% pass^1 on the retail domain but roughly 25% at pass^8 — the same task, eight runs, consistency collapsing. Measuring that against a real counterpart, under real rate limits, is not practical. τ²-bench (June 2025) extends the setup to dual control, where the simulated user also operates tools in the shared environment.

The same simulator serves fault and adversarial testing. Fault injection produces what sandboxes rarely will: 503 bursts, timeouts, slow-drip responses, schema-valid but semantically wrong payloads, rule changes mid-journey. Adversarial testing is now productized: Microsoft's AI Red Teaming Agent (public preview, November 2025) drives more than 20 PyRIT attack strategies, including indirect prompt injection delivered through mock tool outputs, and RAMPART (open-sourced May 2026) encodes such attacks as pytest tests with statistical pass policies. A poisoned counterpart response is precisely what only a simulated counterpart can serve safely.

The Fidelity Gap

A simulator encodes your model of the counterpart — never the counterpart. Undocumented behavior, latency distributions, data quirks, and drift after the counterpart's next release are all invisible in simulation and reappear at integration. A green simulation run verifies the agent against the contract as you understand it. It does not verify the contract, and it does not verify production readiness.

Four rules bound the claim. First, simulation replaces exploratory and regression testing, never certification: conformance for Peppol service providers runs on OpenPeppol's central Testbed, and Germany's OZG-RE invoicing platform mandates its own test environment. Second, before go-live, execute a subset of journeys against the real sandbox or a production pilot. Third, treat every observed divergence between simulator and reality as a defect in the simulator and fix it there. Fourth, do not simulate what you cannot model faithfully — a counterpart with opaque discretionary decisions is simulated only as a guess.

Outlook: Simulated Counterpart Ecosystems

The current cost of the pattern is that every team hand-builds its own simulators. The plausible next step is that counterpart operators ship them: an authority or platform vendor publishing an executable behavioral simulator alongside its OpenAPI specification, versioned together, so the counterpart's semantics become a test artifact. Benchmarks already model both sides of an interaction as tool-using parties in a shared environment; τ²-bench is an early template.

Where both sides of a B2B process are agents, the simulated counterpart becomes a simulated agent, and pre-production testing becomes running both agents against each other in a sealed environment. We expect simulation mode to mature into a standard runtime capability of agent platforms — a first-class switch that swaps counterparts behind unchanged contracts and enforces the provenance labeling described above. Teams that build against simulators from day one will not test faster; they will test things that are otherwise untestable.