Ad Testing: Maximising the impact and ROI of your ads
On this page
Subconscious helps an ad team choose which message to produce before committing the creative budget. Controlled comparisons estimate how specified message changes affect modeled choice, with uncertainty and buyer differences. Use that output to keep the current creative, commission a supported alternative, or leave the decision unresolved when the evidence cannot justify the production cost.
The River City Federal Credit Union case shows why a message comparison can matter. A study with 140 digital twins identified the weakest campaign slogan before spend. The same slogan later performed worst in live Facebook and Instagram testing. That is relevant message-ordering evidence; the case's separate Result Rate change from 0.52 to 0.74 compares campaigns and does not isolate a slogan's sales or ROI effect.
Which creative is worth producing?
Consider a hypothetical art school choosing between two text-only newsletter ads. The school can deliver the advertised class. Both ads occupy the same placement, use the same formatting and link to the same class-times page. Only the headline changes; the complete ads read:
Current ad A
Try a two-hour drawing class
Live online beginner drawing class for adults. $40 for one two-hour session. Bring a pencil and paper; no other materials or fees. Choose an available session before paying. Cancel at least 48 hours before the session for a full refund.
View class times
Proposed ad B
Draw your first still life
Live online beginner drawing class for adults. $40 for one two-hour session. Bring a pencil and paper; no other materials or fees. Choose an available session before paying. Cancel at least 48 hours before the session for a full refund.
View class times
Specify the modeled population and newsletter context before the comparison. Modeled interest in beginner drawing does not establish verified subscribers or actual class buyers. Ask which ad the respondent would choose to open for class details, including the same “open neither” option. Randomize presentation order and hold the offer, destination and surrounding context constant. The contrast estimates the headline's effect on modeled ad choice under those conditions, rather than actual clicks or enrollment.
Before seeing results, the school records the actual quote for copy changes and placement adaptation, the production budget, and the minimum modeled improvement that would warrant that expense. Report B minus A for the declared outcome with an interval that accounts for repeated responses from the same respondent. Check important audience differences and account for any additional creative comparisons. Do not convert a modeled percentage-point difference into dollars of return without a supported link to actual enrollment and contribution.
Commission B if the supported contrast clears the prior threshold and the quoted work fits the budget. Keep A if B does not justify the change. If uncertainty spans actions with materially different consequences, leave the choice unresolved and identify the evidence that could change the production decision. The practical deliverable records the exact selected ad, comparison, uncertainty, cost and action.
Can ad testing really predict sales impact?
Kantar describes sales-linked predictive modeling for ad tests. Such validation can support a prediction without identifying every creative element's causal effect. Randomized exposure can identify an effect on a measured reaction; predicting incremental sales needs relevant outcome evidence.
What do pretest scores support?
System1's current product description distinguishes AI screening from testing with real consumers and covers creative development stages. Inspect the specific test, measured constructs and sales-linked validation, rather than describing all pretests as only liking scores or only finished-ad tests.
What does a platform split test identify?
TikTok's split-test documentation describes mutually exclusive audience groups and holding other variables constant, including when creative is the test variable. Verify the actual assignment and delivery settings. The estimand may be creative under a fixed policy or a creative-plus-delivery policy; algorithmic delivery does not make every randomized platform test nonexperimental.
What does incrementality evidence add?
A suitable randomized holdout can estimate incremental outcomes for its campaign, spend contrast, and time window. A small live pilot can inform a later full-budget choice. It uses real delivery and spending, so it answers a different question from a modeled preference pretest.
How does a randomized experiment on creative work before spend?
Randomly assign concepts or attributes and measure the response the design defines. Choice models can analyze stated choices and relax particular substitution assumptions. ICLV can represent measured latent constructs under additional assumptions; a generated rationale does not prove a psychological mechanism. For a simulated study, inspect the relevant human or market comparison and its fit to the population, task and intended action; an existing comparison can inform that judgment.
The causal-fidelity working paper evaluates estimated-choice-parameter ranks. It does not establish a realized ad-sales effect or human transport for your creative. Request the relevant configuration, comparator and outcome.
Choosing a method: pretest, split test, incrementality, or randomized pre-spend experiment
| Method | What it actually measures | When it runs | Best for |
|---|---|---|---|
| Pretest | Diagnostic or predictive response to creative | During development or before launch | Prioritizing and improving variants under validated scope |
| Randomized platform split test | Performance under specified assignment/delivery policy | Live pilot or campaign | Comparing deployed variants for a defined outcome |
| Incrementality design | Outcomes relative to a counterfactual | Pilot or campaign window | Assessing incremental sales, leads or other supported outcomes |
| Simulated randomized task | Modeled response to assigned alternatives | During development | Choosing a specified creative under a declared production-cost rule and evidence scope |
What to do before your next media buy
Use Subconscious when the immediate commitment is a defined creative choice: compare feasible messages, estimate the specified modeled response and apply a production-cost rule. Kantar and System1 offer sales-linked predictive evidence for their stated tests; a suitable randomized platform test answers a live delivery question. Choose the evidence for the commitment being made.
When the intended claim concerns actual sales or ROI, use relevant live or incrementality evidence with delivery, spend and outcome windows defined. An ROI calculation also needs actual costs and a supported incremental-return measure. See the methods hub and published method evidence.
Book a creative decision review before commissioning the next variant. Bring the complete current and proposed ads, audience, placement, measured outcome, production quotes and evidence already available. Agree which creative decision the comparison must resolve and what would change the action.