See also: First Click Testing, Five Second Test, Spoof Landing Pages
Difficulty: Easy
Evidence strength
Relevant metrics: Conversion
Validates: Desirability
How: Write out your value proposition in 1-3 sentences. Show it to participants for just enough time to read it, then take it away. Finally, ask participants to explain the value proposition back to you in their own words. If the explanation is roughly comparable to your own, count it as a positive result. If 20 people participate, you want 80% to understand your message.
Why: Evaluate whether customers understand a marketing message explaining the value proposition to eliminate a possible false negative bias in later tests where customers indicate they do not want the value proposition when they actually do not understand it. As an example, you will want to run a comprehension test before a spoof landing page test or you won't understand why it doesn't work.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
How to proceed
Before you run a smoke test (that tests commitment), you should run a comprehension test.
You can run the comprehension test manually, showing a piece of paper to the participant for them to read it – and then taking it away to explain what they just read back to you in their own words.
If the participant’s explanation roughly compares to your own, you can count it as a positive result. Otherwise, it’s negative. Aim for a sample size of at least 20 participants and a positive success rate of around 80%. Eighty percent intuitively seems very high - but it must be as regardless of what your value proposition is, your target customers should still understand it.
As a digital alternative to the manual Comprehension Test, the Five Second Test is commonly used.
Who to recruit
Depending on your target market, who you bring to a comprehension test is important.
If your target market has a high school level education, you just need someone with the same basic level of education and vocabulary. At this point, you are not testing to see if someone wants your value proposition, but if they can understand it.
If your value proposition uses industry jargon, you will want to be more critical of who you recruit as test participants. You want to make your value proposition understandable to those who aren’t already converted, so if the customers you want to sell to do not understand your industry specific words, you need to change your value proposition.
Once your target audience understands your value proposition (conversion of at least 80%), you can move on to more sophisticated and (maybe more) expensive tests to probe real customer behavior.
Objective
The goal of this experiment is to ensure that customers accurately understand the product’s value proposition before moving on to further tests. This step is crucial to avoid misinterpretation that could skew results of subsequent experiments, such as a spoof landing page test.
Achieving a high comprehension rate (e.g., 80% of participants accurately understanding the message) indicates the value proposition is effectively communicated and sets a solid foundation for further validation tests.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Examples
HubSpot Academy
In an effort to improve user engagement, HubSpot Academy conducted A/B testing on their homepage hero image. They found that clearer value propositions in their messaging significantly impacted user behavior and conversions. Variant B of their test, which included more vibrant images and an animated “typing” headline, outperformed the control by 6%, leading to an estimated 375 more sign-ups each month.
FSAstore.com
FSAstore.com, an e-commerce company, conducted A/B testing to simplify their website navigation, which initially had too many options and overwhelmed customers. The test compared the current site with a subheader in the navigation to an updated version without the subheader. The result was a 53.8% increase in revenue per visitor, demonstrating the effectiveness of clearer and more concise messaging.
Expoze.io
This company faced difficulties with their website’s homepage readability due to low contrast. They conducted A/B heatmap testing to determine the best design that would improve page navigation and increase attention to specific sections, like the video thumbnail. The new design brought a 25% increase in CTA clicks and over 40% more attention to the desired sections of the homepage.
Related plays
- Comprehension vs. Commitment by Tristan Kromer
- The Real Startup book - Comprehension test by Tristan Kromer, et. al.