Skip to content
AX Framework

Evaluation

Testing Methodology

Task-based testing with real agents, and what to measure.

Inspection tells you what a product provides. Testing tells you whether agents can actually use it. This page describes a lightweight, repeatable method for task-based testing with real agents.

AX proposal

1. Write task scripts

A task script is a goal phrased the way a person would delegate it, plus success criteria:

yaml
id: reorder-coffee
goal: "Reorder the coffee beans I bought last month."
principal:
  account: test-user-07
  limits: { perOrder: 40 EUR }
success:
  - An order exists containing the same SKU as the previous purchase.
  - Exactly one order was created.
  - The agent reported the order ID to the principal.
  - No action exceeded the principal's limits without approval.

Use test accounts and a staging environment. Never test consequential actions against production data.

2. Choose agents

Test with more than one agent if you can, because different agents fail differently. Include at least:

  • A browsing agent that operates the rendered UI.
  • A tool-using agent that calls your API or MCP server.

Record the agent, model, and version for each run so results are comparable over time.

3. Run and record

Run each task several times per agent; agents are not deterministic. For every run, record:

MetricWhat it tells you
Task successDid it meet every success criterion?
StepsHow many requests or actions it took
GuessesDecisions made without explicit information
Errors encountered / recoveredResilience
Human interventionsWhere the agent needed help that wasn't designed in
Unsafe actionsAnything that exceeded limits or duplicated effects
Verified outcomeCould the agent prove what happened?
Cost and timeTokens, requests, and wall-clock time

Guesses are the most informative metric. Ask the agent to log its reasoning, then count decisions based on inference rather than declared information.

4. Analyze

Map each failure and guess to a pillar and a check. Patterns emerge quickly: if most guesses happen while reading product pages, Comprehension is your bottleneck.

5. Retest

After each change, rerun the same scripts. Keep the scripts in version control and treat them like regression tests.

Last updated 2026-10-06. AX Framework documentation describes proposals unless a section is marked as an established standard.