How to tell if your agents are doing good work

Two questions decide whether you can trust an agent with a budget. Can it do something harmful? And are its everyday decisions any good? The first is answered by your rules, which software enforces. The second is answered by results you can read and check yourself.

Rules limit the downside. Results show the upside.

The limits are tested

Spend caps, contact limits, consent, banned phrases, approval limits and the rule that calls experiments are plain code, with no AI in them, and they’re pinned by tests. A change that breaks one fails its test.

The results are public. The cases, the outcome each expects and its latest result are in the test record, and the rules are generated from the code that enforces them.

Results against your target

Each agent’s brief sets a target, such as a cost per first order. Every report leads with how the agent did against it: what it spent, what it got, and what each result cost. The numbers come from your own conversion events, valued by what each is worth to you, not from the ad platforms’ own credit.

The numbers come from code

AI models are unreliable at arithmetic and statistics, so agents don’t do either. Totals, costs per result and confidence come from analytics tools, and the agent explains them.

Tests are called by a fixed rule

When an agent runs an experiment, its length is set before it starts, and the winner is called only at the end, at 95% confidence or more. Until then, you see a labeled reading, and at low volume, “Too early to call”. An agent can’t declare a winner because it likes the early numbers. The arithmetic is in Thirty leads a week.

Holdouts show what email adds

By default, every email send holds back a small group of people who don’t get it. Across sends, comparing the groups shows what email adds beyond the people who’d have bought anyway; when the groups are too small to tell, the report says so.

Every decision is logged

Each action records who proposed it, why, the checks it passed, who approved it and what happened. Each agent’s notebook keeps its hypotheses, decisions and lessons, along with your guidance. When a result surprises you, you can trace it back to the decision that caused it.

You set the pace of trust

Each kind of action has a permission level: Not allowed, You approve, Acts and notifies you, or Acts silently. Start strict, read the reports, and loosen an action’s level once the agent has earned it. If something looks wrong, one switch stops every agent at once.