Also called: Quantitative surveys
See also: Micro Surveys, Net Promoter Score (NPS)
Difficulty: Hard
Evidence strength
Relevant metrics: Jobs to be done ranking, Ranking needs, wants, desires, pains, Net promoter score, Sean Ellis Score
Validates: Viability, Desirability
How: Ask closed-ended questions validating previous suggestions from qualitative research. Avoid ambiguity and false positive results by asking questions about existing customer behavior rather than about opinions and preferences.
Why: They are easy to use, but also potentially misleading. Be careful. Surveys are faceless, without empathy, and assume knowledge of the right question to ask. Plus, you can't ever understand the full context of a written response.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
What should I use closed surveys for?
Closed-ended surveys are often used as part of market validation and general data collection, focusing on gathering structured quantitative data from predefined answer choices. These surveys are particularly effective in exploring the known unknowns, where some level of understanding already exists, but specific, detailed information is needed. They are used when you have a clear direction and aim to fill knowledge gaps.
For instance, a startup founder who has established a problem-solution fit might use closed-ended surveys to prioritize or identify additional issues. The key lies in the structured nature of the questions, which range from simple yes/no queries to more complex matrix questions, allowing for a detailed exploration of specific areas.
The methodology of conducting these surveys can vary significantly, from online forms to in-person interviews, and the time commitment can range from a quick online setup to weeks of in-depth surveys and data entry.
When designing a closed-ended survey, clarity of purpose is crucial. Questions should be directly linked to the actionable outcomes they’re intended to inform, and the type of questions can vary widely, including dichotomous, Likert-type scales, list of items, ordinal, and matrix questions.
After collection, the data should be organized for analysis, employing statistical tools for deeper insights. It’s vital to ensure a sufficient sample size for reliability, particularly within subgroups. Analysis involves looking at overall scores, comparing them to benchmarks, and examining the distribution of responses.
Potential biases such as leading questions, sensitive content, and respondent bias should be carefully managed. Surveys should be designed to be straightforward and allow for responses like “not appropriate” or “don’t know” to capture a more accurate picture of the respondent’s views.
Be wary of false positives: don’t ask for opinions
Asking for opinions, you get opinions. People have opinions on everything, and it doesn’t necessarily reflect what people are actually doing in the real world. Whenever you hear yourself asking question that begin with “Would you…” or “Do you think…”, you will want to stop youself.
It’s better to ask for facts. Questions like “When was the last time you googled…” shows whether potential customers thought about it before you triggered them. If you ask “Would you” or “Do you think”, those same people will be very likely to answer positively. “When was the last time…?” refers to a concrete instance. You want to go for the facts and the instances when you ask.
Don’t force your own perspective
You do not want to focus on the solution because you will be framing the conversation around what you think is important. Instead, you want to go for uncovering the pains and gains of the potential customer.
Instead, you want to first go for playbooks:
Go for situations - concrete instances in the past life of the potential customer you’re asking. “Tell me the last time you thought about…” is a great starter. After that, you can ask follow up questions like “Why did it bother you?”.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Popular tools
The tools below will help you with the Closed-Ended Surveys play.
-
Typeform
Survey tool focused on lowing friction
-
SurveyMonkey
Popular free survey tool
-
Survey.io
Specialized Product-Market Fit survey tool
-
Qualaroo
Context-aware micro-surveys
-
HotJar
Heat-maps, microsurveys, etc. in one package
Examples
LEGO
LEGO’s closed-ended surveys targeted 3,500 girls and their mothers, focusing on play habits. The surveys led to the ‘Friends’ line, featuring vibrant colors and accessories, revolutionizing their approach to inclusivity in play.
Source: How 5 Massive Companies Changed Using Market Research
Apple
Apple utilized online closed-ended surveys through its ‘Customer Pulse’ group, gathering quick feedback on product designs and features. This approach influenced key changes such as larger screens for enhanced user experience.
Source: How 5 Massive Companies Changed Using Market Research
McDonald’s
McDonald’s used closed-ended surveys to understand customer preferences, affecting menu and marketing. Key insights included demand for healthier options, leading to menu adaptations like apple slices.
Source: How 5 Massive Companies Changed Using Market Research
Verizon
Verizon’s market research through closed-ended surveys influenced the strategic decision to allow customers to retain their numbers when switching carriers, enhancing customer satisfaction and loyalty.
Source: How 5 Massive Companies Changed Using Market Research
Related plays
- What Type of Lean Startup Experiment Should I Run by Tristan Kromer