When an ad platform hands campaign decisions to an AI agent, it also reports the results. The only reliable way to know whether the agent caused extra sales is to compare exposed customers with a comparable group that never saw the ads.
- Amazon’s September 29, 2026 Amazon Ads Agent announcement reported more than 25% additional unique customers for advertisers using its AI targeting recommendations, a vendor-reported figure.
- A 2019 Marketing Science study of 15 US Facebook experiments found observational methods often failed to reproduce what randomized experiments measured.
- Google’s Conversion Lift tool supports user-based and geography-based studies but is available only to some accounts, through a Google representative.
- Define the decision and the control group before the campaign launches, because an agent that keeps changing targeting makes after-the-fact comparison unreliable.
Why platform lift needs a second opinion
Ad platforms now run more of the campaign themselves. On September 29, 2026, Amazon Ads renamed its unified buying platform Amazon Ads Agent and described it as powered by agentic AI, with Full-Funnel Campaigns available to all US advertisers.
The announcement also reported that advertisers who applied its natural-language targeting recommendations saw “more than 25% additional unique customers” and more than 10% lower cost per impression on average. These are Amazon’s own figures. We found no sample size or control design published alongside them.
That does not make the claim false. It means the number answers a vendor’s question, which is whether the feature is worth promoting, and not yours, which is whether your next dollar produced a sale that would not have happened anyway. Incrementality, meaning the extra conversions caused by advertising, is the measure that separates those two questions.
What the research says about weak comparisons
The common shortcut is to compare people who saw an ad with people who did not, then adjust for age, interests and past behavior. Research suggests this often fails. Brett Gordon, Florian Zettelmeyer, Neha Bhargava and Dan Chapsky tested it in a paper published in Marketing Science in 2019.
They used 15 US advertising experiments at Facebook, covering 500 million user-experiment observations and 1.6 billion ad impressions, and compared the randomized results with several observational models. The observational methods often failed to match the experiments, even after conditioning on extensive demographic and behavioral variables.
The authors also note that advertising’s effect on outcomes tends to be small, which makes bias easy to introduce. For a buyer, the implication is practical: a dashboard comparison of exposed and unexposed customers is a hypothesis, and a randomized or geographic holdout is the test. The same caution applies to the attribution disagreements we covered in why attribution models disagree.
Three ways to build a control group
A control group is the set of people or places deliberately kept from seeing the campaign. Google’s help page on Conversion Lift defines lift as “the increase in conversions that are caused by the presence of the ad” and describes a treatment group that sees ads and a control group that does not.
| Method | How the control is formed | Main constraint |
|---|---|---|
| Platform user-based study | The platform splits eligible users into exposed and held-out groups | Google says Conversion Lift is available only for some accounts, via a representative |
| Geography-based study | Matched regions get the campaign, others do not | Needs enough regional volume to detect a small effect |
| Campaign split or pause test | Spend is switched off for a set period or region, then results are compared | Seasonality and promotions can distort the comparison |
Google’s page describes both user-based and geography-based Conversion Lift, and notes that the geography type supports offline data. That matters for lead-generation firms whose sales close by phone or in a CRM, because the conversion often happens outside the ad platform.
Running a test, step by step
The order matters more than the tool. Decisions made after seeing results are the main way tests mislead.
- Write the decision the test will inform, such as whether to move budget to the agent-run campaign.
- Choose one outcome that reflects revenue, such as qualified leads or purchases, not clicks.
- Fix the control group, whether users or regions, before launch.
- Keep budget, creative and offers steady in both groups for the whole period.
- Set the run length in advance so it covers your normal buying cycle.
- Compare conversions between groups, divide extra conversions into spend, and decide.
Step four is where AI-managed campaigns create trouble. If the agent shifts targeting daily, the exposed group changes shape mid-test. Ask the vendor how the holdout stays fixed, and see our post on evaluating an agentic marketing vendor for related questions.
When a test cannot answer the question
Small accounts often cannot detect small effects. Because advertising effects tend to be small, as Gordon and colleagues note, a low-volume test can return a result that looks like zero lift when the sample was simply too small to see one. An inconclusive test is not evidence of no effect.
In that case, a longer test, a larger region split or a bigger budget increase is a better response than trusting the platform dashboard. A test also measures one period. Results from a holiday quarter may not hold in January.
Pairing a test with human review is sensible while you build confidence. Our piece on approval gates for agentic ad spend covers how to limit what an agent can change, and our 90-day pilot guide shows where a holdout fits inside it.
Frequently Asked Questions
What is incrementality in advertising?
It is the number of conversions caused by the ads, measured as the difference between people who could see them and a comparable group who could not. Conversions that would have happened anyway do not count.
Can a small business run an incrementality test?
Yes, but the test may be too small to detect modest effects. A geography split or a planned pause on one campaign is usually the most practical option, and results should be read as indicative.
Should I trust the lift numbers an AI ad platform reports?
Treat them as vendor claims until you have a holdout result from your own account. The figures may be accurate in aggregate and still not describe your campaigns.
As ad platforms take on more decisions, the buyer’s job shifts toward proving what those decisions cause. At Grow with Hafsa we build performance-marketing and AI agent programs around a pre-agreed test and baseline, and we are happy to look at how a holdout could work for the campaigns you already run.