Why Per-Test Bayesian Model Loops Stop Scaling
A data science team running many concurrent Bayesian A/B, ABC, and ABCD tests eventually hits the same wall: compute cost and batch runtime scale with the number of tests, not with the amount of insight the tests produce. Left unaddressed, that scaling problem forces a choice between throttling how many experiments run at once or waiting long enough for results that they arrive too late to change a decision.
A documented HelloFresh case study on the PyMC Discourse forum shows how this problem was diagnosed and fixed in production, and is a reference for teams choosing between one independent model per test and a structurally consistent, shared model across test types.
Where the original pipeline broke down
HelloFresh ran large numbers of A/B, ABC, and ABCD test campaigns at once, each fit as its own independent Bayesian model. Diagnosing the existing models turned up strong correlations between some posterior parameters and high autocorrelation in the MCMC chains, both of which slow convergence and cut the effective sample size a team gets for a given compute budget.
That combination of scale and inefficient sampling meant overnight batch runs took long enough that teams either waited past the point where results were still actionable, or limited how many tests they could run concurrently. Neither option scales with test volume.
Fixing the model came before fixing the speed
The team's first move was a correctness pass, not a performance optimization. They proposed a structurally different Bayesian model that removed one parameter compared to the original formulation, used revised priors tailored to the problem, and applied the same structure consistently across A/B, ABC, and ABCD tests.
The new priors also matched domain knowledge: for an ABC test, most prior mass sat on scenarios where the variants' conversion rates were close together, not far apart, while still spreading roughly evenly across the 0–1 range. Under the new model, MCMC chains showed well-behaved mixing, and the correlation and autocorrelation problems that had been slowing convergence were resolved.
Before trusting the new model's output, the team ran parameter recovery simulations: known conversion probabilities were used to generate simulated data, and the posterior distributions were checked against those true values. For an ABC test, the posteriors were correctly centered on the ground truth. That result confirms correct implementation and identifiability under the assumed model; it does not by itself establish calibration or that the model matches the real conversion process.
At this stage, the corrected model alone was already faster: roughly 1.2x for A/B tests and roughly 2x for ABC and ABCD tests. Worthwhile, but the team judged it insufficient given the batch's scale.
Why tuning knobs weren't the answer
Two conventional speed levers were tried next, and both fell short. Cutting MCMC tuning steps from the default 1000 down to 100 saved only about 0.1 seconds per A/B test, not enough at HelloFresh's batch scale. Avoiding model recompilation by defining one PyMC model with data swapped in through pm.Data containers, rather than rebuilding a model object per dataset, improved engineering hygiene but produced a negligible speed gain in practice.
Both attempts optimized how a single test was fit. The bottleneck was somewhere else: how many separate fits the pipeline ran.
The structural fix: one joint model instead of many small ones
The breakthrough was reframing how the batch was fit, not tuning any individual fit further. Instead of looping over datasets and running MCMC separately for each test, the team constructed one large unpooled model that fit all datasets simultaneously.
In this unpooled structure, each test keeps its own statistically independent parameters (nothing borrows strength across tests), but every test shares the same model structure, so the pipeline runs one warmup and adaptation phase and evaluates gradients across all tests in a single vectorized computation instead of restarting the sampler for each one. The pipeline stopped looping over small models entirely; PyMC compiled and sampled one model encoding every test at once.
This is the step that most changed the pipeline's scaling behavior: one larger run replaced many small independent runs carrying the same total data and parameters, and it was measurably more efficient. HelloFresh's own reported runtime figures for this change are documented in the PyMC Discourse discussion; treat them as that team's engineering result, not a Subconscious benchmark.
What generalizes beyond this one pipeline
The unified model also supported two-arm, three-arm, and four-arm comparisons under one consistent prior structure, so A/B, ABC, and ABCD tests could run through the same system instead of separate bespoke pipelines. That consistency simplified both implementation and interpretation, and kept inference accuracy verified by the same parameter recovery approach across test types.
The general lesson transfers past this specific stack: when Bayesian test volume grows, look for the structural bottleneck, how many independent models a pipeline is fitting, before assuming the answer is faster hardware or a lighter-weight per-test model.
Where this fits a causal-testing decision
This case study describes one vendor's engineering work on one customer's pipeline; it is not a Subconscious capability, customer result, or benchmark, and Subconscious does not claim to reproduce this exact pipeline, tooling, or result for any customer. What it illustrates is the argument behind treating causal experimentation as software infrastructure rather than a one-off analysis script: a testing system that has to run at scale needs structural design decisions, not just faster per-test loops. Teams evaluating a causal-testing platform can apply the same standard: see how Subconscious approaches experiment design and what its validated results look like, and where a decision needs grounding beyond a simulated run, pair it with real-human validation on the same causal question rather than treating scale and rigor as a tradeoff.
Limitations
The proof points here, model diagnosis, structural redesign, and parameter recovery validation, come from a single vendor's single-customer engineering case study on the PyMC Discourse forum, not from a controlled study or a Subconscious result. Method boundaries also apply in the other direction: even a well-validated Bayesian pipeline like this one estimates conversion probabilities from the data it's given; it does not substitute for real-human validation when the buyer's decision depends on confirming a causal question with actual people rather than historical conversion data alone.
For general background on the underlying test design, see A/B testing.
