When to Rerun a Causal Experiment Before You Act on It
On this page
A research or insights lead who already ran a causal experiment eventually asks the same question: is the finding still good, or does the budget and strategy riding on it need a second look? A single run is not proof of stability. The Open Science Collaboration replicated 100 psychology studies in 2015. Ninety-seven percent of the original studies had statistically significant results, and 36% of the replications did (Science, 2015). That is reason enough not to treat one experiment as a permanent answer.
That 36% uses one success criterion, statistical significance, on one set of psychology tasks. The same paper reports other criteria that gave different shares. A psychology replication rate is not a forecast of market-simulation accuracy.
What does treating one experiment run as final cost you?
A finding that was never checked for stability can quietly go stale while budget and strategy still ride on it. Market conditions shift, the audience changes, or the inputs that produced the original result stop holding. A rerun doesn't replace the first study; it checks whether the original answer still applies before more budget moves on it.
Three triggers that justify a rerun
Not every finding needs a repeat test. Three conditions do:
Do results stay stable over time?
Consumer preferences and market conditions move. Rerunning the same experiment design after time has passed checks whether the original result still holds.
What happens when the target audience or inputs change?
When the target demographic, product attributes, or pricing strategy shift from what the original study tested, a rerun with those adjusted variables shows how the outcome moves with them.
Benchmark against a human baseline or new data
A recruited human study that uses the same alternatives can check a simulated finding. New comparable human or behavioral data can confirm, revise or reject the earlier result. Define the matching and analysis before making the comparison.
What a rerun looks like in practice
New Age Floral’s engagement involved five iterative experiments, adjusting the question across phases. That illustrates study development, not an exact replication with an unchanged design. An exact replication holds the task and relevant conditions comparable; a sensitivity check deliberately varies assumptions; a new factorial experiment may randomize several attributes. Case study.
Limitations
A rerun checks a specific study's inputs and timeframe. It does not guarantee an outcome, and confidence-interval or uncertainty language only applies where the original study design supports it. A recruited human comparison checks a simulated finding against real participants; it does not turn a causal action test into a usability session or a clinical trial, and it does not prove market performance.
Next step
If a finding is old, or the inputs behind it have changed, a rerun against the new condition is one way to check whether it still holds. Book time to scope a rerun against your original design.
