Idea Validation: Problem

Comprehension Test

Before testing commitment, test comprehension.

Illustration of Comprehension Test
Run a Comprehension Test play

Evidence strength
25

Relevant metrics: Conversion

Validates: Desirability

How: Write out your value proposition in 1-3 sentences. Show it to participants for just enough time to read it, then take it away. Finally, ask participants to explain the value proposition back to you in their own words. If the explanation is roughly comparable to your own, count it as a positive result. If 20 people participate, you want 80% to understand your message.

Why: Evaluate whether customers understand a marketing message explaining the value proposition to eliminate a possible false negative bias in later tests where customers indicate they do not want the value proposition when they actually do not understand it. As an example, you will want to run a comprehension test before a spoof landing page test or you won't understand why it doesn't work.

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

How to proceed

Before you run a smoke test (that tests commitment), you should run a comprehension test.

You can run the comprehension test manually, showing a piece of paper to the participant for them to read it – and then taking it away to explain what they just read back to you in their own words.

If the participant’s explanation roughly compares to your own, you can count it as a positive result. Otherwise, it’s negative. Aim for a sample size of at least 20 participants and a positive success rate of around 80%. Eighty percent intuitively seems very high - but it must be as regardless of what your value proposition is, your target customers should still understand it.

As a digital alternative to the manual Comprehension Test, the Five Second Test is commonly used.

Who to recruit

Depending on your target market, who you bring to a comprehension test is important.

If your target market has a high school level education, you just need someone with the same basic level of education and vocabulary. At this point, you are not testing to see if someone wants your value proposition, but if they can understand it.

If your value proposition uses industry jargon, you will want to be more critical of who you recruit as test participants. You want to make your value proposition understandable to those who aren’t already converted, so if the customers you want to sell to do not understand your industry specific words, you need to change your value proposition.

Once your target audience understands your value proposition (conversion of at least 80%), you can move on to more sophisticated and (maybe more) expensive tests to probe real customer behavior.

Objective

The goal of this experiment is to ensure that customers accurately understand the product’s value proposition before moving on to further tests. This step is crucial to avoid misinterpretation that could skew results of subsequent experiments, such as a spoof landing page test.

Achieving a high comprehension rate (e.g., 80% of participants accurately understanding the message) indicates the value proposition is effectively communicated and sets a solid foundation for further validation tests.

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Examples

HubSpot Academy

In an effort to improve user engagement, HubSpot Academy conducted A/B testing on their homepage hero image. They found that clearer value propositions in their messaging significantly impacted user behavior and conversions. Variant B of their test, which included more vibrant images and an animated “typing” headline, outperformed the control by 6%, leading to an estimated 375 more sign-ups each month.

Source: 11 A/B Testing Examples From Real Businesses

FSAstore.com

FSAstore.com, an e-commerce company, conducted A/B testing to simplify their website navigation, which initially had too many options and overwhelmed customers. The test compared the current site with a subheader in the navigation to an updated version without the subheader. The result was a 53.8% increase in revenue per visitor, demonstrating the effectiveness of clearer and more concise messaging​.

Source: 11 A/B Testing Examples From Real Businesses

Expoze.io

This company faced difficulties with their website’s homepage readability due to low contrast. They conducted A/B heatmap testing to determine the best design that would improve page navigation and increase attention to specific sections, like the video thumbnail. The new design brought a 25% increase in CTA clicks and over 40% more attention to the desired sections of the homepage.

Source: 11 A/B Testing Examples From Real Businesses

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!
Sources

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Ice Breakers

Relieve initial group awkwardness and establish a safe space

Educate

Broaden knowledge or insight regarding the behavior or situation to inform decisions.

Demonstrate

Show practical examples or models of the desired behavior for clear guidance.

Alert

Highlight current actions and their reasons, bringing unconscious habits to awareness.

Train

Develop necessary skills and competencies to enable effective action.

Community events
Product Loop

Product Loop provides an opportunity for Product professionals and their peers to exchange ideas and experiences about Product Design, Development and Management, Business Modelling, Metrics, User Experience and all the other things that get us excited.

Join our community

Made with in Copenhagen, Denmark

Want to learn more about about good product development, then browse our product playbooks.