Why a Flat Choice Model Gets Cannibalization Wrong
A consumer-goods pricing or revenue-growth leader planning a new product launch needs to know one thing before committing trade spend: will this product mostly take share from close substitutes in its own segment, or will it pull incremental volume from across the whole category? Get that wrong and the launch budget goes to the wrong shelf.
The proportional-substitution trap
A basic discrete choice model treats sales as the outcome of many individual purchase decisions. Each product has a utility, driven by attributes like price and shelf availability (All Commodity Volume, or ACV). Utilities convert to purchase probabilities through a softmax function, and the model estimates each attribute's causal effect on sales volume.
p_i = \operatorname{softmax}(u_i) = \frac{\exp(u_i)}{\sum_{j} \exp(u_j)}This structure supports a genuinely causal question: run the model forward with a product's availability set to zero, and the difference between that counterfactual and the observed outcome estimates the sales impact of introducing (or removing) the product.
The trap is in what the flat version of this model assumes about the error term, the catch-all for unobserved reasons a customer picks one product over another (a coupon, a craving). A flat multinomial logit model assumes those unobserved preferences are independent across products. The consequence is mechanical: when a new product enters, the model predicts every existing product loses market share in exact proportion to its starting share.
That is rarely how real substitution behaves. A new kids' toothpaste is far more likely to pull volume from other kids' toothpaste than from an adult whitening line. A flat model that assumes uniform substitution understates cannibalization within a product's own segment and overstates the incremental volume it draws from unrelated competitors, misdirecting launch investment and trade spend.
Nesting products by segment changes the answer
A nested (hierarchical) logit model addresses this by grouping products into nests, typically by segment or brand, that are meant to capture the unobserved factors driving substitution. The nested logit model computes the overall choice probability in two steps: the probability of selecting a nest, then the conditional probability of selecting a specific product within that nest.
P(\text{choose product}) = P(\text{choose nest}) \cdot P(\text{choose product} \mid \text{nest})Within-nest choice probabilities are governed by a shared scale parameter that determines how strongly substitution concentrates among nest members relative to the market as a whole. Structured this way, a counterfactual that removes one product predicts that its nest-mates absorb most of the resulting share shift, while products outside the nest are barely affected. That pattern, close substitutes cannibalized first, is the one launch teams actually see, and the one a flat model cannot produce.
Why the nest structure has to be chosen, not assumed
The nested model's realism depends on getting the nest structure right. An analyst has to decide which grouping (segment, brand, sub-brand, or some hierarchy of these) reflects how consumers actually substitute; the wrong choice still produces a model that misattributes cannibalization, just along a different axis. Building this kind of hierarchical structure to arbitrary depth while keeping estimation numerically stable is itself a nontrivial part of the modeling work, detailed in PyMC Marketing's documentation on nested logit and non-proportional substitution patterns (PyMC Marketing, "Nested Logit and Non-Proportional Patterns of Substitution").
Nest choice is a judgment call that depends on category knowledge, which is also why an automated, always-on cannibalization matrix across an entire portfolio is not a reasonable standing output: a model that silently assumes one nest structure can mislead as confidently as the flat model it replaces. The literature on new-product cannibalization versus market stealing bears this out empirically: evidence from shopping-bot data shows that the two effects behave differently and neither is safe to assume away (Sinha, "New Model Introductions, Cannibalization and Market Stealing: Evidence from Shopbot Data," The Manchester School).
Where this fits a launch decision
Subconscious's causal experimentation can compare launch alternatives: test a proposed product's positioning, price, or segment against holdout conditions and estimate directional causal effects for a defined population before committing capital. A segment-aware substitution matrix answers a narrower question than a general launch comparison, so it should be scoped as its own study design, with the nest structure proposed and checked against category knowledge, rather than treated as a standing report a platform generates automatically.
When the question calls for real-world confirmation, Subconscious can also test or validate studies with real human participants, without changing the underlying causal question. See how this fits into a study design.
Limitations and failure conditions
- A flat discrete choice model works when segments genuinely don't compete for the same occasion, but it silently forces uniform substitution any time they do.
- A nested logit model only fixes this if the nest structure (which grouping, and in what order) matches real consumer substitution behavior; a mis-specified hierarchy still misattributes cannibalization, just less obviously.
- Neither model replaces observed, consumer-level purchase data where it's available. Both are built for the common case in consumer goods: manufacturers who see aggregated product-level sales, not individual purchase records.
- Treat a specific nest structure and its cannibalization estimates as the output of one well-scoped study, not a self-updating matrix across the product line.
Next step
Compare a proposed launch's segment fit and pricing against a defined population using Subconscious's leaderboard of causal effects, or read more on how a study moves from simulated comparison to real-human validation.
