Idea Validation: Problem, Market demand

Closed-Ended Surveys

Survey specific performed behaviors and habits in potential customers

Illustration of Closed-Ended Surveys
Run a Closed-Ended Surveys play

Also called: Quantitative surveys

Difficulty: Hard

Evidence strength
15

Relevant metrics: Jobs to be done ranking, Ranking needs, wants, desires, pains, Net promoter score, Sean Ellis Score

Validates: Viability, Desirability

How: Ask closed-ended questions validating previous suggestions from qualitative research. Avoid ambiguity and false positive results by asking questions about existing customer behavior rather than about opinions and preferences.

Why: They are easy to use, but also potentially misleading. Be careful. Surveys are faceless, without empathy, and assume knowledge of the right question to ask. Plus, you can't ever understand the full context of a written response.

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

What should I use closed surveys for?

Closed-ended surveys are often used as part of market validation and general data collection, focusing on gathering structured quantitative data from predefined answer choices. These surveys are particularly effective in exploring the known unknowns, where some level of understanding already exists, but specific, detailed information is needed. They are used when you have a clear direction and aim to fill knowledge gaps.

For instance, a startup founder who has established a problem-solution fit might use closed-ended surveys to prioritize or identify additional issues. The key lies in the structured nature of the questions, which range from simple yes/no queries to more complex matrix questions, allowing for a detailed exploration of specific areas.

The methodology of conducting these surveys can vary significantly, from online forms to in-person interviews, and the time commitment can range from a quick online setup to weeks of in-depth surveys and data entry.

When designing a closed-ended survey, clarity of purpose is crucial. Questions should be directly linked to the actionable outcomes they’re intended to inform, and the type of questions can vary widely, including dichotomous, Likert-type scales, list of items, ordinal, and matrix questions.

After collection, the data should be organized for analysis, employing statistical tools for deeper insights. It’s vital to ensure a sufficient sample size for reliability, particularly within subgroups. Analysis involves looking at overall scores, comparing them to benchmarks, and examining the distribution of responses.

Potential biases such as leading questions, sensitive content, and respondent bias should be carefully managed. Surveys should be designed to be straightforward and allow for responses like “not appropriate” or “don’t know” to capture a more accurate picture of the respondent’s views.

Be wary of false positives: don’t ask for opinions

Asking for opinions, you get opinions. People have opinions on everything, and it doesn’t necessarily reflect what people are actually doing in the real world. Whenever you hear yourself asking question that begin with “Would you…” or “Do you think…”, you will want to stop youself.

It’s better to ask for facts. Questions like “When was the last time you googled…” shows whether potential customers thought about it before you triggered them. If you ask “Would you” or “Do you think”, those same people will be very likely to answer positively. “When was the last time…?” refers to a concrete instance. You want to go for the facts and the instances when you ask.

Don’t force your own perspective

You do not want to focus on the solution because you will be framing the conversation around what you think is important. Instead, you want to go for uncovering the pains and gains of the potential customer.

Instead, you want to first go for playbooks:

Go for situations - concrete instances in the past life of the potential customer you’re asking. “Tell me the last time you thought about…” is a great starter. After that, you can ask follow up questions like “Why did it bother you?”.

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Popular tools

The tools below will help you with the Closed-Ended Surveys play.

  • Typeform

    Survey tool focused on lowing friction

  • SurveyMonkey

    Popular free survey tool

  • Survey.io

    Specialized Product-Market Fit survey tool

  • Qualaroo

    Context-aware micro-surveys

  • HotJar

    Heat-maps, microsurveys, etc. in one package

Examples

LEGO

LEGO’s closed-ended surveys targeted 3,500 girls and their mothers, focusing on play habits. The surveys led to the ‘Friends’ line, featuring vibrant colors and accessories, revolutionizing their approach to inclusivity in play.

Source: How 5 Massive Companies Changed Using Market Research

Apple

Apple utilized online closed-ended surveys through its ‘Customer Pulse’ group, gathering quick feedback on product designs and features. This approach influenced key changes such as larger screens for enhanced user experience.

Source: How 5 Massive Companies Changed Using Market Research

McDonald’s

McDonald’s used closed-ended surveys to understand customer preferences, affecting menu and marketing. Key insights included demand for healthier options, leading to menu adaptations like apple slices.

Source: How 5 Massive Companies Changed Using Market Research

Verizon

Verizon’s market research through closed-ended surveys influenced the strategic decision to allow customers to retain their numbers when switching carriers, enhancing customer satisfaction and loyalty.

Source: How 5 Massive Companies Changed Using Market Research

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Ice Breakers

Relieve initial group awkwardness and establish a safe space

Educate

Broaden knowledge or insight regarding the behavior or situation to inform decisions.

Demonstrate

Show practical examples or models of the desired behavior for clear guidance.

Alert

Highlight current actions and their reasons, bringing unconscious habits to awareness.

Train

Develop necessary skills and competencies to enable effective action.

Community events
Product Loop

Product Loop provides an opportunity for Product professionals and their peers to exchange ideas and experiences about Product Design, Development and Management, Business Modelling, Metrics, User Experience and all the other things that get us excited.

Join our community

Made with in Copenhagen, Denmark

Want to learn more about about good product development, then browse our product playbooks.