Home › Deterministic Test Data Specifications for AI Coding Agents

Test Data Belongs in the Specification

A field report on deterministic test data, AI coding agents and system-level verification

A field employee saw four appointments. The dispatcher saw no corresponding tour.

Both views looked plausible in isolation. Together they described two different systems.

The implementation was wrong.

We found this while building a field-service scheduling prototype with coding agents. The application covers customer bookings, employee availability, planning, payments, cancellations, notifications and process history. It uses Flutter for three user roles, FastAPI, PostgreSQL and asynchronous workers.

The individual functions were not particularly difficult. The failures existed between persistence, process transitions, background execution and role-specific projections.

Local tests passed. Endpoints responded. The connected business process was still inconsistent.

We did not solve that by writing a longer prompt.

We changed the environment in which the agent worked. Instead of developing against an almost empty database and constructing fixtures alongside each feature, the agent received a versioned, deterministic application state defined separately from the implementation under review.

Test data became part of the specification.

This article is for architects and senior engineers introducing coding agents into stateful, relational business applications. It does not propose a new testing discipline. Integration testing, Test Data Management, executable data specifications and deterministic data generation are established practices.

The narrower question is more useful:

What changes when a coding agent can materialise and inspect a reviewable test data specification before it declares an implementation complete?

By the end of this article, you should be able to decide when a separate test data specification is justified, how it differs from local fixtures and how to keep the agent’s implementation separate from the acceptance verdict.

Coding agents compress an existing risk

Human developers have always been able to encode the same incorrect assumption in implementation and tests. Coding agents intensify that risk because they compress several activities into one reasoning context.

Conventional workflow	Compressed agent workflow
Requirements may be interpreted by one person and verified later by another	One agent interprets, implements and verifies in one session
Fixtures may predate the implementation	Fixtures are often generated to support the current implementation
Review introduces another context	The first independent review may happen only after all artefacts agree
Contradictions may emerge during handover	One blind spot can propagate through code, fixtures and assertions

A typical loop looks like this:

Requirement
→ Implementation
→ Fixtures
→ Tests
→ Green

Everything can be internally consistent and still be wrong.

Unit tests and mocks remain necessary. They answer focused questions about functions, contracts and error handling.

They do not automatically prove that a paid appointment:

enters planning exactly once;
appears for the customer;
becomes visible to the dispatcher;
reaches the assigned employee;
survives retries without duplication;
reaches a terminal business outcome.

Those properties cross persistence, APIs, asynchronous execution and user-role projections.

The agent needed constraints and observations that were not derived solely from the implementation it had just written.

An empty database could not provide them. A small fixture created for one endpoint usually could not provide them either.

A shared declarative review surface

We placed executable DATAMIMIC XML models beside the API and acceptance testing specifications. The repository contains separate but connected artefacts:

specs/api/openapi.yaml
    External HTTP contract

specs/acceptance/*.feature
    Gherkin behaviour and expected outcomes

specs/data/<scenario>/*.xml
    Versioned executable data specifications

Each artifact has a different responsibility.

OpenAPI defines the exposed operations.

Gherkin defines observable behaviour.

The DATAMIMIC XML model defines the entities, relationships, lifecycle states, constraints and deliberate variations required to execute that behaviour.

The running application produces the evidence.

A behaviour change can therefore update the contract, acceptance scenario and required data state in one review.

Test data is no longer a private implementation detail hidden inside a test function. It becomes a versioned test data specification describing the conditions under which the behaviour must work.

The actual authoring flow

This project uses DATAMIMIC CE 4.1.0 and an XML-first repository model.

The Authoring API is invoked with a transient AuthoringSpecV1 request. It validates the requested structure and expectations, returns verified=true and produces executable XML.

No model.dm.json or verification.json files are committed.

The versioned XML descriptor is the durable model source used for PostgreSQL materialisation:

Transient AuthoringSpecV1 intent
→ Authoring API validation
→ verified=true
→ executable XML
→ reviewed XML in the repository
→ PostgreSQL materialisation

This is a project-specific choice.

The Authoring API currently supports File and Memstore targets. The complete PostgreSQL path therefore uses DATAMIMIC’s Raw XML database support with <database> and nested <generate> elements.

The human reviewer and the agent still work with the same declarative language. The durable artefact in this repository is XML rather than a checked-in JSON intent model.

Why we used a DSL

We could have generated the same data with Python, SQL, YAML, a fixture library or other test data generation tools.

DATAMIMIC was not the only technically possible choice.

We selected it because we needed several properties together:

relational generation across the application schema;
deterministic replay;
named business scenarios;
assertions about generated state;
versioned and diffable models;
structured validation;
an authoring surface usable by coding agents;
one artefact that humans and agents could both inspect.

A Python script can generate complex data. The problem is not expressive power.

The problem is reviewability.

In general-purpose code, the intended business state is often distributed across loops, helper functions, persistence calls and randomisation logic. A reviewer must mentally execute the generator to understand the condition it creates.

Why are only three employees qualified for rope access?

Which request states are represented?

Which payment is expected to fail?

Which process must reach a terminal state?

A declarative model moves these decisions closer to the surface. It focuses on what must exist rather than every technical step required to produce it.

The repository, for example, contains explicit lifecycle models:

<state-machine id="terminanfrageLifecycle" start="eingegangen">
    <transition from="eingegangen" to="in_pruefung" weight="0.9"/>
    <transition from="eingegangen" to="zurueckgezogen" weight="0.1"/>
    <transition from="in_pruefung" to="rueckfrage" weight="0.2"/>
    <transition from="in_pruefung" to="alternative_angeboten" weight="0.2"/>
    <transition from="in_pruefung" to="zur_planung" weight="0.4"/>
    <transition from="in_pruefung" to="erledigt" weight="0.1"/>
    <transition from="in_pruefung" to="abgelehnt" weight="0.1"/>
</state-machine>

<generate name="terminanfrage_verlaeufe" count="120">
    <key name="status" generator="terminanfrageLifecycle"/>
</generate>

This is not application code. It is a versioned statement about the states and transitions the generated data may contain.

A contract test compares selected XML transition edges with the backend domain transitions. The data model does not become the behavioural authority, but drift between the declared data state and the backend model becomes observable.

A DSL is not universally superior.

Python remains appropriate for small unit-test objects, technical utilities and cases where the generation mechanism itself is under test.

A practical decision rule is:

Use code when setup is a local test detail. Use a declarative model when the state itself is part of the behaviour under review.

The declared application state

The seed pipeline starts with a freshly migrated PostgreSQL 17 database.

DATAMIMIC writes the generated domain data directly to PostgreSQL. The Python wrapper only injects temporary runtime credentials, invokes DATAMIMIC and reads quality metrics afterwards. It does not generate or insert business records itself.

The integrated descriptor populates 27 application tables, including:

one tenant;
25 employees and three dispatchers;
300 cleaning objects;
customers, contracts and cleaning orders;
25 territories and 300 assignments;
four planning runs and process instances;
process steps and append-only step events;
300 appointments and twelve appointment requests;
100 field-service jobs;
incidents and appointment relationships;
timeline events;
outbox events;
idempotency records;
mobile synchronisation cursors.

Fourteen named scenarios cover conditions such as qualification scarcity, part-time capacity, approved leave, recurring contracts, failed payments, cancellations and asynchronous retries.

The number of records is not the quality metric.

The structure is.

Focused acceptance scenarios

The repository contains an executable Gherkin specification for planning:

# language: de
Funktionalität: Planungslauf und messbarer Vergleich

  Szenario: Equipment bleibt eine harte Nebenbedingung
    Angenommen Seilzugang-Objekte und genau drei qualifizierte Mitarbeiter
    Wenn Gebiete und Tagestouren geplant werden
    Dann ist jedes Seilzugang-Objekt einem qualifizierten Mitarbeiter zugeordnet

  Szenario: Vergleich und Baseline sind reproduzierbar
    Angenommen unveränderte Eingabedaten und ein fester Clock- und Random-Seed
    Wenn derselbe Lauf zweimal berechnet wird
    Dann sind Baseline, Vorschlagskennzahlen und Verbesserung identisch

The focused equipment scenario uses exactly three qualified employees.

The larger F1 planning comparison uses five qualified employees. These are different scenarios designed to test different properties and should not be conflated.

The model does not decide how many qualified employees constitute a correct business policy. A domain expert still owns that decision.

It proves something narrower:

The generated state contains the constraint that the scenario claims to represent.

Random data adds volume.

Specified data adds decision-relevant variance.

One appointment, three realities

The clearest product-level observation was an appointment that existed differently depending on who looked at it.

The field employee saw four appointments. The dispatcher saw no corresponding tour.

Each screen was locally plausible.

The connected system was globally inconsistent.

Testing the employee screen alone would not expose that. Testing the dispatcher screen alone might not expose it either.

The relevant invariant was cross-role:

The same confirmed appointment must have compatible representations in the customer, dispatcher and employee views.

That rule does not belong to one widget or one endpoint. It belongs to the business system.

The available evidence proved that the role-specific projections were inconsistent during review. It did not prove whether the technical cause was a missing transition, a stale projection or a query defect.

There is currently no automated acceptance scenario that loads all three roles and asserts the same appointment identity across them. The iOS integration tests exercise the roles sequentially, but not as one cross-role invariant.

That distinction matters.

This case is evidence for the need to add such an invariant. It is not evidence that an automated DATAMIMIC test already detected the defect.

The next permanent regression scenario should verify at least:

appointment identity
assigned employee
time window
object identity
customer identity
active versus historical visibility

across all three role projections.

The data model did not diagnose the root cause.

It created a stable review environment in which the contradiction became visible and could be preserved as a future regression class.

A passing gate beside an omitted requirement

The most technically useful result came from the F1 planning comparison.

The scenario contained:

300 objects;
25 employees;
six weighted Hamburg districts;
four-, six-, eight- and twelve-week service cycles;
the complete 24-week least-common-multiple horizon;
1,320 contractual services;
seed 20260728;
a versioned district-weighting file.

Both planning strategies scheduled the same 1,320 services. Neither left appointments unplanned.

Baseline strategy

The baseline:

retained the original cycle phases;
sorted objects by postcode;
preserved stable input order inside each postcode;
assigned employees round-robin;
created routes using nearest neighbour from the employee depot;
resolved equal distances by object ID.

Candidate strategy

The candidate used:

greedy cycle alignment;
capacity- and equipment-constrained territory partitioning;
OR-Tools daily routes;
residual planning from Monday to Friday.

The pipeline itself is explicit:

Load data
→ Resolve cached distance matrix
→ Partition territories
→ Align cycles
→ Calculate daily routes by territory
→ Aggregate results
→ Calculate baseline comparison
→ Persist proposal

Under the same deterministic scenario, approximated drive time fell from 2,110,175 seconds to 214,337 seconds.

That is a reduction of 89.84 percent.

Our acceptance gate required an improvement of 8 percent.

The measurement used Haversine distance multiplied by 1.35 at a constant 30 km/h. It did not use real road geometry, traffic, parking times, roadworks or holidays.

The absolute travel times are therefore deterministic approximations. The relative comparison between both strategies is the meaningful result.

The omitted property

In the same run, the workload-spread indicator increased from 790.11 to 2,823.44 service minutes.

The indicator is the population standard deviation of the total assigned service duration per employee over the complete planning horizon:

σ = sqrt(sum((xᵢ - mean(x))²) / N)

where xᵢ is the sum of soll_dauer_min assigned to employee i.

The test passed.

We had defined an acceptance threshold for drive time and none for workload distribution.

The correct conclusion is not that the candidate scheduler contained a proven defect.

The acceptance specification was incomplete. It constrained travel time but remained silent about acceptable workload balance.

The workload metric itself also has an important limitation: it is not normalised by each employee’s available capacity, part-time schedule or approved absences.

It detected a substantial distribution change. It was not yet a sufficient fairness criterion.

A better product-level acceptance property could use:

assigned minutes divided by available minutes;
maximum overload relative to capacity;
percentile spread of relative utilisation;
a Gini coefficient over capacity-normalised workload.

The correct metric remains a domain decision.

The test-data scenario created the stable environment in which the omission became measurable.

DATAMIMIC did not diagnose the missing requirement. It materialised the complex state needed to expose its consequence.

Reproducing the planning result

The measured comparison is versioned and executable:

.venv/bin/pytest tests/acceptance/test_f1_measurement.py -q
.venv/bin/python scripts/measure_f1.py

The runtime used:

DATAMIMIC CE: 4.1.0
Seed:          20260728
Python:        3.12.10
OR-Tools:      9.15.6755
Platform:      Darwin 25.5.0 arm64

The F1 acceptance test verifies:

due appointments:       1,320
unplanned appointments: 0
repeat runs:             identical results

The complete DATAMIMIC verification path is:

make datamimic-check
make seed

make datamimic-check verifies:

the exact CE package pin;
the public Authoring API result with verified=true;
XML linting for all scenarios;
bounded dry-runs and the explicitly documented row cap.

make seed executes only the PostgreSQL XML descriptor.

Two seed runs for business date 2026-07-29 produced the same aggregate hash across all 27 application tables:

3e9741523fd08e0f5ddd44505f5081230e38e65e91e6e519fbc03726698faa82

This proves reproducibility for the pinned project configuration.

It does not prove that every possible environment or generator is automatically deterministic.

From model to observable evidence

The generated state enters the actual application architecture.

PostgreSQL remains the business source of truth.

Domain changes and their corresponding outbox events are written transactionally. TaskIQ workers process reconstructable references. Persisted idempotency and resumable process states handle duplicate delivery and retries.

Redis Streams is the default broker in the local environment.

RabbitMQ can be enabled as an alternative transport. In that mode, each queue has a dead-letter queue, while Redis remains available for the TaskIQ result backend.

Neither broker owns the booking, payment or planning state.

The same business object can be observed through:

DATAMIMIC XML model
→ PostgreSQL state
→ FastAPI response
→ TaskIQ process history
→ Flutter role interfaces

The three Flutter variants share code and the generated OpenAPI client, but have separate entry points, iOS schemes, display names and bundle identifiers.

This is where the platform becomes a review surface rather than only a data generator.

The agent can execute the declared scenario, call application interfaces, run the acceptance and integration gates and inspect the resulting views inside the isolated local environment.

The repository enforces tenant scope through explicit MandantId parameters and PostgreSQL Row Level Security as a final protection layer.

This case study does not claim that coding agents receive unrestricted production database access. Production authentication and production payment approval are outside the prototype scope.

The agent is not the final authority.

Its explanation remains a proposal.

Independent gates provide the verdict:

OpenAPI contract validation;
domain and state-machine contract tests;
DATAMIMIC model checks;
PostgreSQL integration tests;
process tests for terminal outcomes and idempotency;
Flutter tests on an iOS Simulator;
repository hygiene and artefact checks.

Property-based testing can explore additional values and event sequences inside these scenarios.

Contract-testing tools can protect interfaces between components.

Database snapshots can provide fast, fixed starting points.

These techniques complement the materialised state. None is a universal replacement for the others.

The important property is separation.

The agent may help author the model. The XML remains reviewable, the acceptance properties exist outside the implementation, and observable system behaviour determines whether the feature is accepted.

Determinism is not coverage

For supported seeded, rule-based generators, the same DATAMIMIC engine version, model and seed produce the same generated output.

In this project, complete replay also requires pinned:

external resource files;
database migrations;
business date;
broker initialisation;
worker configuration;
clock configuration.

ML-based generation targets statistical fidelity rather than byte-identical replay.

Determinism makes failures attributable.

The agent can change one implementation detail and rerun the same difficult state. A changed result can be attributed to the implementation instead of fixture drift.

But a perfectly repeatable scenario can still be shallow.

Determinism is a control variable, not a coverage strategy.

We use two complementary layers.

Named deterministic scenarios provide stable regression evidence and reproducible handover.

Controlled variation changes cardinalities, boundaries, scarcity and lifecycle combinations to explore more of the state space.

Randomness alone is insufficient. Random data can be diverse and irrelevant at the same time.

The goal is controlled complexity against declared properties.

When does an executable data specification make sense?

Not every application needs one.

Small fixtures remain sufficient when the state is local, synchronous and cheap to reconstruct.

A separate declarative model becomes useful when several of these signals appear:

Signal	Local fixtures may be sufficient	Executable data specification becomes useful
Relationships	Few and local	Multiple aggregates and projections
Processes	Short and synchronous	Asynchronous or multi-step
User roles	One view	Several views of the same object
Lifecycle	Few states	Retries, failures and terminal outcomes
Setup cost	Cheap to recreate	Expensive or error-prone
Handover	One developer	Humans, agents and CI must reproduce it
Review	Code explains the state	The state itself carries business meaning

The threshold is not a fixed number of tables.

The decision turns on whether the state must be reviewed independently from the code that consumes it.

A practical adoption path

Teams do not need to model their complete platform on day one.

Start with one workflow whose correctness crosses system boundaries.

1. Select one connected process

Choose something such as:

payment
→ booking
→ assignment
→ role visibility

Avoid starting with the complete domain.

2. Declare the relevant state

Define:

entities;
relationships;
lifecycle states;
deliberate conflicts;
expected invariants;
terminal outcomes.

3. Keep behaviour and data separate

Use Gherkin or another acceptance format for expected behaviour.

Use the data model for the conditions in which that behaviour must hold.

Do not make the generator the behavioural oracle.

4. Materialise the real persistence state

Use the actual database and service boundaries for the decisive integration test.

An in-memory substitute may still be appropriate for faster lower-level tests.

5. Add one cross-boundary invariant

For example:

A confirmed appointment has the same identity, assigned employee and time window in every role.

One useful invariant is better than twenty generic assertions.

6. Make the environment replayable

Pin:

model;
seed;
database migration;
external resources;
clock or business date;
broker initialisation;
worker configuration.

7. Keep the verdict independent

The coding agent may execute the gates and analyse failures.

It must not redefine the success criteria while correcting the implementation.

What changed in practice

We did not run a controlled experiment.

This was one prototype. The team also understood the domain better in month two than in month one. We cannot isolate the effect of executable data specifications from every other improvement in the development process.

Two effects were concrete enough to name.

Findings became transferable

A defect could be handed to another developer or agent as a named scenario and seed instead of a verbal description of a local environment.

The planning comparison can be reproduced with the same XML model, seed and pinned weighting file.

Review preparation decreased

The reviewer starts from a known scenario, a visible business outcome and an independent test result.

Recreating customers, bookings, payments and process histories manually is no longer the first step of the review.

These are observations from one project, not universal productivity claims.

The relevant metric for us was the distance between a wrong assumption and an observable contradiction.

That distance became shorter.

What the approach does and does not do

Data specifications do not decide whether a business policy is correct.

They make the selected policy explicit and testable.

The model can contain the same incorrect assumption as the written specification. The agent can still misunderstand the requirement.

Twenty-seven populated tables do not prove completeness. Fourteen scenarios do not prove sufficient coverage. Generated data is not production data.

The responsibilities remain clear:

humans define and accept the business policy;
specifications describe required behaviour;
Gherkin expresses observable examples;
DATAMIMIC materialises the required data conditions;
the agent implements, executes, observes and corrects;
independent gates and the running system provide the verdict.

Declarative artefacts make decisions reviewable.

They do not make those decisions correct.

Bottom line

This is not a new testing discipline.

It is a field report on what changed when we made the data state a first-class, declarative artefact in a coding-agent workflow.

An agent inside a specified and populated system can discover that its solution does not work.

A human who can inspect the model, state and outcome can verify the contradiction and investigate its cause.

FAQ

How do you test code written by AI coding agents?

Declare the data conditions and expected properties separately from the implementation. Testing AI-generated code this way materialises those conditions in the real system and uses independent contract, integration, process and UI tests to judge the result.

Is this different from ordinary integration testing?

The final verification still uses integration and system tests. The difference is that the data conditions are maintained as a separate, versioned specification instead of being privately assembled inside each test.

Why use a DSL instead of Python or SQL?

You do not always need a DSL. We chose one because the data state carried business meaning and had to remain inspectable by humans, coding agents and CI across implementation, testing and review.

Does deterministic test data provide sufficient coverage?

No. Determinism provides reproducibility. Coverage requires multiple scenarios, controlled variation and explicit acceptance properties.

Try the workflow

DATAMIMIC Community Edition provides the XML DSL, deterministic data generation engine, CLI, Python API and agent authoring surface used in this case study.

A practical starting point is to inspect a versioned model, run make datamimic-check and reproduce one bounded scenario before integrating the approach into a larger workflow.