Why AI agent evaluation starts with the evidence
This is an AI agent evaluation method rather than a product launch: every claim the workflow made had to survive an independent critic before we trusted it. This was an internal Autonomous engineering project for a workflow that researches Shopify storefronts and prepares relevant outreach. Its core requirement was evidential: a statement about a store should be supported by something the workflow actually observed there.
Earlier experiments had produced plausible claims that needed substantial correction. Investigation showed that problems could originate in the surrounding software as well as the model’s wording. Giving the model a stronger prompt would not repair a pipeline that handed it an incorrect interpretation of the page. The build therefore treated evidence handling as an application responsibility.
Give the critic a different job from the builder
The person who builds a component is the worst person to decide it works, so a separate critic had to break it first. Work was divided into components with written interfaces. A builder implemented each component; a separate critic tried to break it and returned reproductions rather than a general opinion. The builder then addressed the cause and submitted the result to another round.
This creates useful tension when the first implementation looks convincing. The critic is looking for a counterexample: a statement that passes without evidence, a state transition that can happen twice or a supposed learning result that disappears under a better test. The record retains failed rounds, so the final result can be assessed alongside the problems that shaped it.
Workflow: Written contract → Builder implementation → Adversarial test → Reviewed correction.
Make decisions traceable across the workflow
If the system stops halfway through an action, it has to work out on restart what actually happened rather than doing it twice. A shared ledger tracks progress and records state changes. Updates are coordinated so two workers do not casually advance the same item independently. External provider calls are treated as a particular source of uncertainty: a local process can stop after an external action succeeds but before the local result is saved.
The design records intent and reconciles uncertain outcomes on restart. This is relevant beyond outreach. An ecommerce integration can face the same gap after creating a fulfilment request, updating a CRM or asking another system to perform an action. Recovery behaviour needs to be designed around that boundary, not assumed from the happy path.
Test evidence and learning separately
We tested two different things separately, whether a claim was actually supported by what the workflow saw, and whether the learning component really learned anything. The build uses two independently implemented evidence checks. One checks whether support exists in the fetched material; another evaluates the proposed claim against that support. Agreement is required, and a later approval is not supposed to reverse an earlier rejection.
Replay tests examine the learning component using cases where it should promote a change, refuse it or continue collecting information. The critic challenged whether the optimiser could mistake noise for a winner. A separate cached set of twenty-five real Shopify storefronts supported repeatable workflow tests. These stores were test fixtures, not client references, and none was contacted for the test.
A method demonstrated, not a production result
The build produced a review record and concrete changes prompted by adversarial testing. It did not establish increased replies, booked calls or revenue. The reviewed state still contained unresolved defects, and a full automated CI gate was not in place for this internal repository. Those limits rule out describing it as a finished unattended sales engine.
What transfers to a store automation engagement is the review method: define the contract, test claims against evidence, challenge recovery behaviour and require approval for changes to learned behaviour. AI-assisted development is useful when the team can explain what it tested and what remains uncertain, not just show how quickly code appeared.
People behind the work
Questions and answers
What does a useful adversarial review produce?
A reproducible failure against the agreed requirements, with enough detail for the builder to investigate. The goal is to change the implementation or expose an unresolved limit, rather than add a second reassuring summary.
What makes the critic independent?
It is a separate review role tasked with finding counterexamples and reproducing failures against the agreed contract. The builder’s confidence is not the acceptance criterion.
Why use cached storefronts in testing?
Recorded pages make tests repeatable and avoid confusing changes in a live website with changes in the implementation. Fixture brands are not presented as clients or endorsers.
How can this method help Shopify automation?
It can challenge whether an action has adequate evidence, whether retries repeat an external action and whether proposed optimisations respect the business rules. The particular rules and acceptance cases are scoped to the store.
What does AI agent evaluation look like in practice?
A written contract for each component, a builder who implements it, a separate critic who returns reproducible counterexamples rather than an opinion, and a record that keeps the failed rounds. On this build it also meant two independently implemented evidence checks that both had to agree, replay tests for the learning component, and human approval before any learned behaviour changed.