Why a 20% Retention Rate Means Different Things for a 10-User Cohort and a Million-User Cohort
A marketing analytics lead looks at a dashboard and sees a young cohort retaining at 20%. The number reads the same whether it came from 10 users or 1 million. It isn't the same number. A ratio hides how much confidence you should place in it, and that gap is where teams greenlight a channel or a campaign off a signal that was never statistically load-bearing, then reallocate budget when the next cohort regresses to a different rate.
Why a retention ratio is the wrong thing to model
Retention is always between zero and one, and it's a quotient: active users divided by cohort size. Two cohorts can post the same 20% and carry completely different amounts of information.
| Cohort | Cohort size | Retention rate | How much that number tells you |
|---|---|---|---|
| A | 10 (2 active users) | 20% | Almost nothing. One more or fewer active user swings the rate by 10 points |
| B | one million users | 20% | A stable estimate. The same swing barely moves the rate |
Treating those as interchangeable collapses the uncertainty that separates them. The fix is to stop modeling retention directly and instead model the count of active users with a likelihood, so retention becomes a scored estimate rather than a raw division.
Modeling active users instead of a ratio
A binomial likelihood ties the number of active users to cohort size and an underlying retention probability:
N_active ~ Binomial(N, p)where N is cohort size and p is the retention probability the model estimates rather than assumes. A link function then relates p to cohort age, elapsed time, and seasonality. This structure produces a credible interval around every retention estimate, not just a point value, so a 20% rate from 10 users comes back with a wide interval and a 20% rate from a million users comes back tight.
A companion revenue model follows the same logic: revenue per cohort-period is modeled with a distribution tied to the number of active users and an average-revenue-per-user parameter, so retention and revenue share the same cohort-level uncertainty rather than being reconciled after the fact.
What this buys forecasting, specifically
Juan Orduz's Bayesian treatment of this approach, built on a binomial likelihood for retention paired with Bayesian Additive Regression Trees (BART) for the non-linear feature interactions, reports that credible intervals widen for smaller and younger cohorts while the same pooled structure still produces forecasts for very young cohorts with little data of their own (Orduz, "Cohort Revenue & Retention Analysis: A Bayesian Approach," arXiv:2504.16216). That's the practical payoff: a model that tells you when to trust a young cohort's number and when to wait for more data, instead of a dashboard that reports every ratio with equal confidence.
Where this discipline generalizes
Quantifying uncertainty explicitly, rather than reporting a point estimate and moving on, is the same standard Subconscious applies before treating a result as a decision input: an estimate is only as useful as the confidence interval attached to it. It isn't a claim that any specific cohort pipeline runs inside Subconscious's product.
Limitations and what this doesn't prove
This is a modeling-methods explainer, demonstrated on a synthetic dataset with known ground truth so the model's accuracy could be checked against it. It's a useful pattern for anyone building cohort-level CLV forecasts, not a validated production benchmark, and it doesn't establish results in a live, messy revenue dataset. Extending it, by layering in acquisition channel as a covariate, pooling across hierarchical markets, or swapping BART for a neural-network component, is a reasonable next step for a team with the engineering capacity to build and maintain it, not a guarantee of the same interval widths or forecast accuracy on a different business.
Next step
If the open question is whether a channel or campaign result is real or just a young cohort's noise, Subconscious's leaderboard shows how causal estimates get reported with their own uncertainty, and how Subconscious works walks through the same estimate-with-confidence-interval discipline applied to a decision before it ships.
