Nine months into an engagement the hardest question is also the easiest one to answer badly. Is this working? The dashboard says yes. Revenue is up. They are responsive, the reporting arrives on time, and the account is tidy. None of that is evidence.
Not because agencies are dishonest, though some are. Because growth and an agency's contribution to growth are two different quantities, and almost every reporting system in this industry is built in a way that blurs them.
Working means something specific
There is only one definition of working that survives contact with arithmetic: what would be different if the work stopped?
That sounds abstract until you count how much reported performance fails it. Branded search converts beautifully and collects people who typed your name. Retargeting closes carts that were going to close. A record December arrives in a category that has a record December every year. All three appear in the monthly report as success, and all three would have happened largely without you.
So the question is not whether the numbers went up. It is how much of the rise the agency caused. That figure is harder to obtain, always smaller than the attributed one, and the only one worth paying for.
Four tests that separate the two
1. Turn it off somewhere
A holdout is the only test that settles the argument. Hold back a matched set of geographies, or pause one campaign type across half your market for three or four weeks, and compare the two halves. It costs you some revenue inside the window. That cost is the price of knowing, and it is materially lower than the cost of another nine months of not knowing.
If your agency has never run one, has no plan to, and responds to the suggestion by explaining why the read would be contaminated, you have learned something useful. The objections are often technically correct and always conveniently timed.
2. Reconcile to the bank
Total up platform-reported revenue across every channel, then compare it to what actually landed in the bank, net of returns, cancellations, shipping and payment fees. The platforms will always sum to more than the bank, because several of them claim the same conversions. The number worth watching is not the size of the gap, it is the trend in it. If reported return holds steady while the gap widens, performance did not improve. The counting did.
3. Count new customers, not revenue
Blended revenue growth can come from new buyers, from existing buyers purchasing more often, or from a price increase. All three are good for the business and only the first means acquisition is working. Ask for first-time customer counts, or first-time sold jobs, by month, with no revenue figure attached. It is the least gameable number in the business and, in most engagements, the one least likely to appear in the reporting.
4. Watch cohort quality, not just cost
Cost per acquisition can hold perfectly flat while the people being acquired get steadily worse. Compare the cohort acquired nine months ago against last month's cohort at the same age: ninety-day repeat rate, close rate, average order value, refund rate. If cost is stable and cohort quality is sliding, you are not holding performance. You are working through a finite pool and approaching the bottom of it.
Things that look like working and are not
- Return on ad spend improving while spend falls. Cutting the top of the funnel always flatters the average, because what remains is the demand that was closest to converting anyway. It reverses within about two quarters.
- A record month in a month when the entire category had a record month. Check last year's same week, a category index, or what competitors are hiring for before accepting the credit.
- More conversions immediately after a tracking change. The counter changed, not the result. Those two are indistinguishable in a dashboard and trivially distinguishable in the bank.
- A high volume of tests with no decisions attached to them. Testing earns its cost only where a losing result switches something off. Otherwise it is activity wearing the vocabulary of research.
- A tidy account. Tidiness is hygiene. It correlates with competence and proves nothing at all about contribution.
Things that look like failure and are not
- Cost per acquisition rising while total contribution rises. If the next increment of spend still returns more than it costs, a higher average cost is the correct result of scaling rather than a problem to be fixed.
- A flat month immediately after measurement gets cleaned up. Honest numbers look worse than dishonest ones on the day they arrive. The comparison you want is against corrected history, not against the old reporting.
- Fewer leads and more sold work. Somebody stopped optimising toward volume. That is the job.
- A test reported as a failure. An agency that tells you a hypothesis was wrong, with the number, is an agency whose claims of success are worth something.
The tells that are not metrics at all
After the arithmetic there are four behavioural questions, and in practice they discriminate faster than the data does.
- Have they ever told you to spend less? An agency paid a percentage of spend that has never once recommended reducing it is not advising you. It is billing you.
- Do they bring bad news before you find it? The order is the whole signal. Everybody has a bad month. Only some tell you in week two rather than at the quarterly review.
- Can anyone there state your contribution margin, within a point or two, without looking it up? If not, every optimisation decision is being made against a number they do not know.
- When they were wrong, did they say so with a figure attached? Not that the approach was adjusted. That this was expected to be worth a specific amount, it was worth nothing, and here is what got switched off.
When it genuinely cannot be proven
Sometimes the honest answer is that the data will not settle it. Spend too small for a clean holdout, a sales cycle longer than the period under discussion, a market that moved underneath everybody at once. Arguing about the last nine months in that situation produces nothing except a worse relationship.
The move is to stop arguing backwards and start pre-registering forwards. Agree now, in writing: the next test, what result counts as a win, what result counts as a loss, how long it runs, and what each side does in either case. The following review becomes a reading rather than a debate.
A good agency will be relieved by that conversation. They have spent the engagement being judged on a number they know is inflated, and a pre-registered test is their first opportunity to be credited with what they actually caused. The reaction you get when you propose it will tell you most of what you were trying to find out.