Bayesian A/B Testing at Scale: Why Millions of Observations Slow MCMC Down
A head of experimentation running an A/B test with millions of observations faces a real trade-off: full Bayesian inference swaps a single point estimate for three richer outputs, namely a Bayes factor, a Region of Practical Equivalence (ROPE) analysis, and a 95% credible interval, but a naive Markov chain Monte Carlo (MCMC) model becomes impractically slow once a test passes roughly a million rows. The team can re-architect the sampling pipeline to keep compute flat, or it can get the causal answer before the test ever needs a million live observations.
Why observation count breaks a standard MCMC model
At every MCMC step, the model evaluates the likelihood of each observation given the current parameters: 100 observations means 100 evaluations per step, 1,000,000 observations means 1,000,000, with sampling time scaling roughly linearly with observation count when the number of steps is held constant. Removing group indexing (splitting the data into separate vectors per group instead of indexing into a shared parameter array) makes the model run more than 3x faster at 1 million observations, yet the underlying scaling problem remains: both the indexed and non-indexed models keep slowing down as the dataset grows.
The histogram approximation: bin observations instead of evaluating each one
The fix is a planning-example technique from Bayesian statistics, not a Subconscious product feature: group all observations into a set number of bins (say, 500), then evaluate the likelihood only at each bin's center, weighted by how many observations fall in that bin. Rather than run 1 million likelihood evaluations at every MCMC step, the model now needs just 500 bin-center evaluations, a roughly 2,000x reduction in per-step likelihood evaluations at that observation count. That ratio describes the likelihood-evaluation count, not overall runtime: fixed per-step costs such as gradient evaluation, NUTS tree-building, and compilation dominate once the likelihood term is cheap, so measured runtime speedups are much smaller (75 seconds to 13 seconds, about 5.8x, at 500,000 observations). Because compute scales with bin count rather than observation count, a dataset of 1 million or 100 million observations requires the same number of evaluations per step. Accuracy improves with more bins as the approximation converges toward the continuous case, so bin count becomes the accuracy-versus-compute dial.
In one documented run, a binned model built this way sampled 100 million observations in 22 seconds on a standard iMac, with total runtime including model compilation at roughly 30 seconds, versus a linear increase in runtime for the default (non-binned) model at that scale. Posterior uplift estimates from the binned model tracked closely with the default model across simulated true uplifts ranging from 0.1 to 1.0 at 500,000 total observations: the default model averaged 75 seconds per run at that volume, the binned model averaged 13 seconds, with no meaningful loss of inferential precision reported. The technique extends to multi-group A/B/C/D tests but does not generalize to regression models with continuous predictor variables.
What this means for the underlying decision
Binning solves the "MCMC compute grows with observation count" problem for a team willing to build and maintain that engineering. It does not solve a different, earlier problem: even a binned sampler still means the test may need to run against millions of real users to resolve small effects, and until posterior probability or ROPE monitoring supports early stopping, a bad variant remains live, though ramped rollouts limit how much of that exposure is at full scale.
\text{Speedup} \approx \frac{N_{\text{observations}}}{N_{\text{bins}}}Subconscious addresses that earlier problem instead of the sampling-speed problem. It runs controlled experiments on a simulated population and estimates causal effects with confidence intervals describing sampling variability within that simulation, before a change reaches a single real customer; external validity to the real-customer population is a separate, unquantified assumption. Our best configuration reaches 87% of the measured human ceiling on one study: 0.832 rank correlation against the published human result, where two independent samples of real humans reach 0.959. Across all 43 studies that pass design filters the mean is 0.73. It is a validation result, not a guarantee for a new market, per the causal fidelity paper. When a decision depends on it, a team can extend the same causal question into real-human validation on the same test without re-deriving the question from scratch.
Limitations
Subconscious does not implement or endorse the histogram-binning MCMC technique described above; that is a general Bayesian-statistics engineering pattern for scaling a live pipeline, not a Subconscious capability. And running a decision-quality simulated test first does not remove the need to monitor a live rollout at scale once the change ships. The two are complementary steps in the same pipeline, not substitutes for each other.
Further reading
For teams that do need to scale live Bayesian MCMC pipelines, the binning approach traces back to established large-N approximation methods, including "Light and Widely Applicable MCMC: Approximate Bayesian Inference for Large Datasets" and "Informed Sub-Sampling MCMC: Approximate Bayesian Inference for Large Datasets". The open-source PyMC library remains a common tool for building these models.
See how this same class of causal question runs on a simulated population, before it ever needs live-test scale, on Subconscious's research page or the leaderboard of tested effects. To scope a specific test, book time with the team.