Problem

Comprehension Test

Before testing commitment, test comprehension.

Illustration of Comprehension Test

Strength of evidence
25

Relevant metrics: Conversion

Validates: Desirability

How: Write out your value proposition in 1-3 sentences. Show it to participants for just enough time to read it, then take it away. Finally, ask participants to explain the value proposition back to you in their own words. If the explanation is roughly comparable to your own, count it as a positive result. If 20 people participate, you want 80% to understand your message.

Why: Evaluate whether customers understand a marketing message explaining the value proposition to eliminate a possible false negative bias in later tests where customers indicate they do not want the value proposition when they actually do not understand it. As an example, you will want to run a comprehension test before a spoof landing page test or you won't understand why it doesn't work.

This card is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

How to proceed

Before you run a smoke test (that tests commitment), you should run a comprehension test.

You can run the comprehension test manually, showing a piece of paper to the participant for them to read it – and then taking it away to explain what they just read back to you in their own words.

If the participant’s explanation roughly compares to your own, you can count it as a positive result. Otherwise, it’s negative. Aim for a sample size of at least 20 participants and a positive success rate of around 80%. Eighty percent intuitively seems very high - but it must be as regardless of what your value proposition is, your target customers should still understand it.

As a digital alternative to the manual Comprehension Test, the Five Second Test is commonly used.

Who to recruit

Depending on your target market, who you bring to a comprehension test is important.

If your target market has a high school level education, you just need someone with the same basic level of education and vocabulary. At this point, you are not testing to see if someone wants your value proposition, but if they can understand it.

If your value proposition uses industry jargon, you will want to be more critical of who you recruit as test participants. You want to make your value proposition understandable to those who aren’t already converted, so if the customers you want to sell to do not understand your industry specific words, you need to change your value proposition.

Once your target audience understands your value proposition (conversion of at least 80%), you can move on to more sophisticated and (maybe more) expensive tests to probe real customer behavior.

Get a head start

Download the Comprehension test sheet

The test comprehension sheet forces you to think through your value proposition before putting time and effort into testing it. Do you explain the pains and gains of your product and do you back it up with evidence?

Grab the 'Comprehension test sheet'
Download comprehension test sheet

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Sources

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Idea Validation playbooks