Thirty leads a week: when can you call a test?
At 30 leads a week, a test of a modest change can’t be called within a month. This essay works through the arithmetic, the rule agents use to call a test, and what they do while they wait.
Heraldic · Written
A company that sells through a sales team runs search ads that bring in about 30 inquiries a week. About 3% of clicks become an inquiry. The team builds a new landing page and wants to know whether it’s better, so the ads split traffic between the current page and the new one.
The rule that calls a test
Agents don’t ask a language model whether a test worked. The statistics are a few lines of ordinary code, the same for every agent, and the model only reads the answer. For each version, the code computes a rate, conversions divided by exposures, then runs a one-sided test of two proportions. The result, shown as “confident”, is one minus the p-value: how surprising the gap would be if the two versions were really the same. It is not the probability that the new version is better, and the product never calls it a chance.
- Until every version has at least 20 conversions, the test reads “Too early to call”, whatever the rates say.
- With enough data but under 75% confidence, it reads “No clear difference yet”.
- From 75%, it names the leader and the confidence: “New page ahead · 82% confident”.
- A winner is called only when the planned run ends, at 95% confidence or more.
- A test that reaches the end of its run without a winner concludes “No difference”, and the current version stays.
Week one: nothing to report
At 15 leads per version per week, neither version reaches the 20-lead minimum in the first week. The first reading of any kind comes in the second week.
Week three: a real lift, and still no call
Suppose the new page really is 20% better, and after three weeks the results land exactly there: 45 leads on the current page from 1,500 clicks, and 54 on the new one. By the rule above, that is 82% confidence. The new page is shown as ahead, but it isn’t called the winner: if the two pages were the same, a gap that size would still turn up about one time in five.
The same three weeks with bigger gaps, run through the same formula:
| New page leads | Rate | Lift | Confidence | What it says |
|---|---|---|---|---|
| 54 | 3.60% | +20% | 82.1% | New page ahead · 82% confident |
| 58 | 3.87% | +29% | 90.4% | New page ahead · 90% confident |
| 68 | 4.53% | +51% | 98.6% | Called: over the 95% bar |
| 72 | 4.80% | +60% | 99.5% | Called: over the 95% bar |
Called once, at the planned end
Confidence moves every day. If you check daily and stop the first time it crosses 95%, random noise gets fourteen chances to look like a winner. A simulation of 40,000 fourteen-day tests of two identical versions, at 500 clicks a day each, shows the cost. Stopping on the first day over 95% picked a false winner about one time in six. Looking once, at the end, picked one about one time in twenty. The simulation is a short script you can run with Node, and it prints the same numbers every time.
So every test has a planned length before it starts, and the call is made once, when that length is reached. The daily figure is labeled as a reading, never as a verdict. You can stop a test early, but a stopped test names no winner.
How long a 20% lift takes to find
To plan a test’s length, statisticians use a standard formula for how many exposures each version needs to detect a given lift. It answers a different question from the rule above: how long to run so that a real lift of a given size has a good chance of being called. Here it is for one-sided 95% confidence and 80% power, meaning a real lift of that size is called four times out of five.
| Real lift | Clicks per version | Leads on the current page | Weeks |
|---|---|---|---|
| 20% | 10,958 | about 329 | 21.9 |
| 50% | 1,981 | about 59 | 4.0 |
| 100% | 588 | about 18, under the 20-lead minimum | 1.3, set by the minimum |
At 30 leads a week, reliably finding a 20% improvement takes about five months, longer than most landing pages stay unchanged. A 50% improvement takes about a month, and even then one real lift in five won’t be called. That isn’t a flaw in the rule. It’s all that 30 leads a week can tell anyone.
“Too early to call” is a result. It’s the one that stops you moving money on noise.
What agents do with small numbers
Low volume doesn’t mean no testing. It means testing differently, and being clear about what is and isn’t known.
- They plan a test’s length before it starts, show where it stands, and call it only at the end. A free-shipping test on day 3 of 21 that shows 6.0% against 4.8% still reads “Too early to call”. A gift-set test on day 9 of 14 reads “Reading so far: Roast date ahead · 87% confident”, and isn’t called.
- They test differences big enough to find: a different offer, page or audience, not a different button color. At 30 leads a week, only lifts of around 50% or more can be found within a month.
- They steer by a step with enough volume. An agent steers by the deepest step of its funnel that had at least 100 conversions in the last 30 days. Thirty inquiries a week is about 129 in 30 days, which is enough. Ten qualified leads a week is about 43, which isn’t. So the agent steers by inquiries and reports on qualified leads while the signed contracts arrive weeks later.
- They close tests cleanly. A test that runs its full length without a winner keeps the current version, and the agent’s notebook records that no difference was found, so the next plan doesn’t repeat it.
- They leave the arithmetic to software: rates, confidence and the 20-conversion minimum are calculated, and the agent interprets the result and writes the report.
The same rule at a store
The rule doesn’t change with the business. The pace does. A coffee roaster with daily orders ran a 14-day test and got a clear answer: “Roast dates won: 37% more first orders per click, 95% confident. For brewing gear alone there was no difference.” A lead-generation company running the same rule on 30 leads a week would still be in its first month. That’s why each agent’s pace is set by its funnel, and why a report at low volume says “too early to call” more often than anything else.