Should You Run a Fake Door Test, or Test the Concept First?
On this page
A product owner deciding whether to build a feature needs evidence about a specific commitment. A fake door measures real clicks on an offer. A concept comparison can examine an offer before implementation. Subconscious can help make the pre-build comparison useful: define complete alternatives, assign tested framings across modeled respondents, and return a comparison with uncertainty for a named product decision. Choose the method from the unresolved question, existing evidence, and the cost of being wrong.
A public example shows why a comparison can challenge an internal preference. In the River City FCU case, 140 digital twins tested messages before a live campaign. The weakest modeled slogan also performed worst live. That agreement concerns message ordering in that project. The separate reported campaign improvement does not isolate a slogan effect or prove adoption of an unbuilt feature. For a feature decision, establish whether the task, population, and available validation support the proposed use.
What does a fake door click rate actually measure?
A fake door exposes an unbuilt feature offer to real users and measures clicks. Amplitude and Chameleon describe live entry points, measurement, and disclosure when a feature is unavailable. Targeting and randomized copy can be part of the instrumentation. Define eligible exposure and unique clicks before reading a rate; repeated clicks can reflect confusion. Clicks alone do not establish later adoption.
The entry point also creates an expectation. Explain the feature's status after a click, avoid implying an available purchase, and decide what happens to any follow-up contact information. Repeatedly disappointing current customers can cost trust. Reuse existing analytics and feature controls when available, rather than build a new testing system for one question.
Running a controlled experiment before you build anything
A concept study can randomly assign framings and compare stated adoption intent before product code is built. In a simulated task, the endpoint is modeled intent. Prespecify exposure, assignment, the estimator, and supported uncertainty; inspect calibration and relevant human evidence before relying on the result.
| Fake door click test | Controlled concept experiment | |
|---|---|---|
| What it measures | Actual clicks under the exposure conditions | Modeled or human stated adoption intent under an assigned task |
| Segment detail | Cohort targeting where configured | Segment definition and coverage must be planned |
| Preparation cost | Design, copy, exposure, and instrumentation | Stimuli, audience definition, assignment, analysis, and review; may need no product code |
| Information requirement | Plan visitor exposure, allocation, and power for the click comparison | Plan task sampling, segment coverage, and precision for the intended estimate |
| What the result leaves unresolved | Clicks alone do not explain motivation or establish adoption; follow-up methods can investigate | Stated or modeled intent does not observe live clicks or later adoption |
Concept intent, live clicks, and later adoption answer different questions. Both randomized concept tasks and live tests can estimate effects of specified variants under their own designs. A modeled response does not become an observed click because the same label appears in both tasks. A live click does not establish repeat use or willingness to keep paying.
A worked choice: concept evidence or a live entry point?
Consider fictional Deskline, a project workspace for small teams. The current plan costs $29 per workspace per month before tax and includes five users, project lists, due dates, CSV export, and email support. The proposed calendar add-on costs another $5 per workspace per month, billed with the monthly plan. Calendar access lasts through the paid month after cancellation; cancellation prevents the next add-on renewal. The add-on would show existing project due dates in a read-only calendar. Calendar editing, reminders, external calendar sync, and managed services are excluded.
The product owner must decide by Monday whether to allocate a fictional $1,200 implementation budget to the calendar entry point and disclosure flow. That figure is an internal production cost, not a Subconscious research fee or a promised return. Existing support requests suggest a scheduling problem, but do not establish demand for this exact paid add-on. If existing records already settle the intended commitment, use those records without commissioning another comparison.
For an unresolved offer question, the owner can scope a modeled comparison among current Deskline project leads at teams of five or fewer users. Confirm that evidence supports modeling that customer context. Each respondent sees the full current plan, add-on scope, price, renewal and cancellation terms, plus the same image and response options. Randomly assign only the headline:
- A: “See next week's project deadlines in a calendar.”
- B: “Find project deadlines by day.”
The task asks whether the respondent would add the calendar at the next renewal or keep the current plan. The outside option is the current project list at the current $29 price. Preserve the complete offer and outside option across headlines. Record assignment, repeated-response handling, the planned contrast, and how uncertainty will be estimated. The output concerns modeled intention under that task, not observed add-on purchases or a forecast of subscription revenue.
Before testing, the product owner and analyst agree what difference would matter, what uncertainty and comprehension problems would prevent reliance, and what evidence would justify spending the production budget. Include the comparison's actual cost, available time, relevant calibration, and consequences of a false positive. A small, uncertain headline difference may not justify any implementation change. Where matched evidence is missing and the decision is costly, scope the additional evidence needed rather than treating a generated sample as customer validation.
What do the results of a concept experiment tell you?
Possible result patterns and checks include:
- A useful difference across tested framings. Inspect uncertainty, comprehension, calibration, and relevance to current customers. The result may select wording for the next bounded step. It does not guarantee click or adoption gains.
- A mixed or inconsistent difference. Inspect segment coverage and whether wording changed expectations about the feature. A headline that implies reminders would not truthfully represent Deskline's read-only calendar.
- No difference or a negative estimate. Inspect uncertainty, baseline appeal, and comprehension. Equally appealing headlines can have no difference. A simulated rejection alone does not justify retiring the feature.
The decision attachment names the accepted headline, complete offer, endpoint, uncertainty, relevant existing records, and remaining gap. A product owner can retain the current list, approve a precisely scoped live entry point, request a specific correction, or leave the question unresolved. A result without enough support must remain unresolved. A positive result cannot silently authorize building the entire feature.
Moving from a concept study to a live test
For Deskline, a live entry point could offer a preview of the proposed calendar add-on. The disclosure must clearly state that the feature is not available; no purchase or charge occurs. The product owner must approve the truthful entry and disclosure words, eligible cohort, exposure allocation, observation period, measurement, and any follow-up data handling before launch. The analyst defines a click denominator and a useful-click criterion in advance. Existing records may support that design; new human research depends on the material evidence gap, rather than a mandatory sequence.
The delivery owner receives the exact accepted entry text, complete offer, unavailable-feature disclosure, cohort, instrumentation requirements, spending limit, and deadline. The product owner signs the bounded action. Reopen the decision if price, functionality, population, destination, or claim changes, or if later clicks and customer behavior contradict the premise. Inspect later use and retention separately before treating the add-on as adopted.
A live test can contradict a modeled preference. When matched human evidence is needed, confirm actual participants, recruitment, exposure, measurement, and analysis; adapting a simulated task may require design changes. Review the study workflow for the comparison's documented scope. Other providers also run controlled experiments; compare actual designs and evidence, rather than assume a capability boundary from a product category.
Choose the next commitment with Subconscious
Subconscious's value in this decision is an inspectable comparison tied to the product owner's next action: a complete offer, a defined modeled endpoint, an uncertainty statement, and the conditions for using the result. Inspect the research evidence against the task and intended claim. Benchmark performance does not establish this feature's demand. Live clicks require actual exposure and instrumentation; adoption requires later behavior.
Book a Subconscious decision review with the feature proposal, complete offer and current alternative, existing customer evidence, unresolved question, implementation cost, deadline, and accepting product owner. Define the comparison and permitted commitment before allocating the next build budget.
