Every agency now says it runs AI agents. The useful question for a buyer is not whether a vendor uses agents, but how it proves the agents work before they touch your budget, your site or your brand.
- A Digital Applied survey of 250 agencies (fielded January to March 2026) found 41% had at least one agent in production, with median reported ROI of 3.2 times.
- The bottom quartile reported 0.7 times, below break-even, and the publisher linked that outcome to missing workflow-level evaluation setups.
- The publisher’s billing check on 47 agencies found self-reports overstated ROI by about 18% and understated token spend by about 24%.
- Ask any vendor to show its evaluation harness, its approval gates and its cost per task before you agree to scope.
What the survey found
Digital Applied surveyed 250 independent marketing and development agencies across the US, EU and APAC between January 14 and March 7, 2026. Agencies had between 8 and 180 staff. The survey reported that 41% had at least one agent in production, up from 9% in the first quarter of 2025, while 58% were still piloting.
Returns were uneven. Median reported ROI was 3.2 times, the top decile reported 11 times, and the bottom quartile reported 0.7 times, which is below break-even. Median token spend, meaning the usage fees paid to AI model providers, was $1,800 a month per active agent.
For a buyer, the spread matters more than the median. Agents are not reliably profitable for the agencies running them, so a vendor’s confidence is not evidence. Your question is whether this vendor sits near the top or the bottom of that range.
What an evaluation harness is
Evaluation harness: a fixed set of test cases with known good outcomes that an agent is run against before launch and after every change. It turns “the agent seems fine” into a measurable pass rate.
The survey’s publisher reported that bottom-quartile agencies almost always lacked a workflow-level harness, and that agencies which built evaluations first reported ROI around 11 times. The report gives no correlation coefficient or subgroup sizes, so this is an association and not a proven cause. It also named evaluation and testing complexity as the top blocker, cited in the top three by 49% of agencies.
In plain terms, a harness for an SEO agent might hold 50 past pages with known issues and check whether the agent finds them without inventing new ones. If a vendor cannot describe its version of this, it is probably judging results by eye.
Weak and strong vendor answers
The table contrasts typical answers to the checks that matter most. It is a judgment aid built from the survey’s themes, not a scored standard.
| Check | Weak answer | Strong answer |
|---|---|---|
| Testing | “We review outputs as we go” | A named test set, a pass rate and a re-test after each change |
| Human approval | Agent publishes or spends automatically | Defined approval gates for spend, publishing and client-facing changes |
| Cost | One flat fee, usage not mentioned | Cost per task and a monthly usage cap |
| Reporting | Activity counts such as pages produced | Business outcomes tied to your baseline |
| Explainability | “The model decided” | A log showing what the agent did and why |
Explainability is not a side issue. Client trust and explainability was the second most cited blocker in the survey, named by 37% of agencies, and cost predictability the third, at 32%.
Questions to ask, in order
Use these in a first call. Each one is designed to be hard to answer with a slide deck.
- What tasks will agents perform for us, and what will a human still approve?
- Show us the test set used to check that work, and the latest pass rate.
- What does one task cost in usage fees, and what is the monthly cap?
- What did the last agent failure look like, and how was it caught?
- How will you measure success against our own baseline, not an industry average?
Approval gates deserve special attention for paid media. Our analysis of approval gates for agentic ad spend explains why spend should never be fully unattended. For the wider vendor-selection picture, see how to choose an AI marketing agency.
How much weight the survey deserves
Treat the numbers as indicative. The sample was not random: recruitment came from the publisher’s own database, a partner list and cold outreach, and the publisher says it over-indexes on agencies already considering agentic AI, so true adoption may be 5 to 10 points lower.
The ROI figures are also self-reported. The publisher compared billing data for 47 agencies and found self-reports overstated ROI by about 18% and understated token spend by about 24%. That is a useful warning for buyers: ask vendors for billed numbers, not claimed ones. The same logic applies to our guide on what agentic SEO actually is, where autonomy claims need the same scrutiny.
Frequently Asked Questions
What is an evaluation harness in AI marketing?
It is a fixed set of test cases with known correct outcomes, used to score an AI agent before launch and after each change, so quality is measured instead of assumed.
Is the 3.2 times ROI figure a reliable benchmark?
Only as a rough guide. It is a self-reported median from a non-random survey of 250 agencies, and the publisher found its own sample overstated ROI by about 18% where billing data existed.
Should a vendor’s agents ever act without human approval?
Low-risk, reversible tasks can run unattended. Spending, publishing and client-facing changes should keep a human approval step until the vendor shows a track record on your account.
Choosing a vendor is easier when you ask for proof of testing and not a demo. At Grow with Hafsa we design AI agent and performance-marketing work with clear approval gates and measurable baselines, and we are happy to pressure-test a proposal you are already weighing.
Once you have chosen a vendor, test it before committing: our guide to piloting an AI marketing agent over 90 days sets out the scope and exit criteria.