Evaluation
Testing Methodology
Task-based testing with real agents, and what to measure.
Inspection tells you what a product provides. Testing tells you whether agents can actually use it. This page describes a lightweight, repeatable method for task-based testing with real agents.
AX proposal1. Write task scripts
A task script is a goal phrased the way a person would delegate it, plus success criteria:
id: reorder-coffee
goal: "Reorder the coffee beans I bought last month."
principal:
account: test-user-07
limits: { perOrder: 40 EUR }
success:
- An order exists containing the same SKU as the previous purchase.
- Exactly one order was created.
- The agent reported the order ID to the principal.
- No action exceeded the principal's limits without approval.Use test accounts and a staging environment. Never test consequential actions against production data.
2. Choose agents
Test with more than one agent if you can, because different agents fail differently. Include at least:
- A browsing agent that operates the rendered UI.
- A tool-using agent that calls your API or MCP server.
Record the agent, model, and version for each run so results are comparable over time.
3. Run and record
Run each task several times per agent; agents are not deterministic. For every run, record:
| Metric | What it tells you |
|---|---|
| Task success | Did it meet every success criterion? |
| Steps | How many requests or actions it took |
| Guesses | Decisions made without explicit information |
| Errors encountered / recovered | Resilience |
| Human interventions | Where the agent needed help that wasn't designed in |
| Unsafe actions | Anything that exceeded limits or duplicated effects |
| Verified outcome | Could the agent prove what happened? |
| Cost and time | Tokens, requests, and wall-clock time |
Guesses are the most informative metric. Ask the agent to log its reasoning, then count decisions based on inference rather than declared information.
4. Analyze
Map each failure and guess to a pillar and a check. Patterns emerge quickly: if most guesses happen while reading product pages, Comprehension is your bottleneck.
5. Retest
After each change, rerun the same scripts. Keep the scripts in version control and treat them like regression tests.
Related
Last updated 2026-10-06. AX Framework documentation describes proposals unless a section is marked as an established standard.