Also called: Sean Ellis Test, "40% Test"
See also: Closed-Ended Surveys, Net Promoter Score (NPS)
Difficulty: Easy
Requires existing audience or product
Evidence strength
Relevant metrics: Jobs to be done ranking, Ranking needs, wants, desires, pains
Validates: Desirability
How: If over 40% of customers say they would be very disappointed, there is a product/market fit. Allow for open-ended responses to understand why. If you are close to 40%, learn what product or segment tweaks can get you above the line and what solutions customers consider as viable alternatives to their problems.
Why: While a product/market fit survey provides an excellent way to gauge customers' dependency and fondness for an existing product, it does not validate actual product/market fit. If you have to ask whether you have product/market fit, the answer is simple: you don't. Product/market fit surveys can give false positive results Ð especially in the early stages of a product.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
Scale when it’s time
The timing of when you decide to scale your idea can make or break your business. Scale too early, and users will leave. Scale too later, and a competitor will beat you to the punch.
To help determine when it is time to push ‘GO’, the Product-Market fit survey invented by Sean Ellis, can come in handy.
Ask a simple question…
The product-market fit (PMF) survey asks the customer one very specific question:
How would you feel if you could no longer use [this product]?”
Let participants answer with one of these mulitple choice responses:
- Very disappointed
- Somewhat disappointed
- Not disappointed (it isn’t really that useful)
- N/A – I no longer use [product]
By comparing nearly 100 startups, Sean Ellis, who came up with the survey format, found that if over 40% of users responded that they would be “Very dissapointed” to stop using the product, there’s a great chance that the solution had found its Product-Market fit. He found that those companies that scored below 40% all struggled to reach traction.
And follow up…
Receiving a PMF score is a great metric to gauge your progress on your road to scale. However, if you don’t follow up with more qualitative questions or more experiments, it can be hard to understand exactly what impacted your score to move from one plateau to the other.
Typical follow-up questionns
The followinng can serve as inspiration to what kind of questions you might ask your customres. As with all research, if you can find this evidence looking at real behavioral data you will want to do that. Real evidence, observing what people do, will always trump opinions and what people say. You build better products by asking better questions.
- Please help us understad why you selected this answer. (Open ended question)
- Have you recommended this product to anyone? (No, Yes: ___)
- What type of person do you think would benefit most from the product? (Open ended question)
- How can we improve the product to better meet your needs? (Open ended question)
- How often do you use our product? (Never, Once a year, A couple of times a year, Once a month, A couple of times a month, Once a week, Every day)
- How would you feel if you could no longer use our product? (Very disappointed, Somewhat disappointed, No reaction / neutral)
- Have you ever recommended us to friends or family? (Yes / No)
- What do you think sets us apart from our competitors? (Open ended question)
- What could we improve on? (E.g. Customer Support, Reliability of Service, Design, Functionality, Other)
- Could you tell us a bit about yourself? What’s your employment status? (Open ended question)
- How did you discover/come across this product? (Blog, Google/Search, Twitter, LinkedIn, Facebook, Word of mouth, other)
- What would you likely use as an alternative if this product were no longer available? (I probably wouldn’nt use an alternative, I would use an alternative: ___)
- Would it be okay if we followed up by email to request a clarification of one or more of your responses? (Yes - enter email / No)
How does Product-Market Fit look like?
“If you have to ask whether you have Product/Market Fit, the answer is simple: you don’t.” – Eric Ries
You ultimately want to put yourself in a position where you can observe real user behavior that proves that you have indeed found Product-Market fit.
How many responses do I need?
Buffer conducted a study using the PMF survey and found that they only needed 40-50 respones for the results to carry significance. The critical factor is ensuring a diverse range of respondents and understanding their characteristics. It is essential to avoid surveying customers who are not invested in your product.
To prevent skewed results, it is imperative to collect feedback from users who are actively engaged with your product or company, possess a basic understanding of its core features, and have used your product within the two weeks preceding the survey. This approach guarantees fresh and relevant insight.
It is worth noting that the “Sean Ellis Product Market Fit Survey”, also referred to as the “40% Test,” calls for a 40% response rate after distributing the survey to the designated customer group.
Sean Ellis recommends to survey a crowds with the following characteristics:
- People that have experienced the core of your product offering
- People that have used your product at least twice
- People that have used your product in the last two weeks
When should I send Product Market Fit surveys?
While identifying the appropriate individuals to survey is important, determining the best time to send out the survey is equally crucial. Should the survey be sent during the product development phase? Following a sales pitch? Or, perhaps, during the early hours of the morning?
Product Market Fit during product development
The answer to this question varies depending on the stage of your business and the specific insights you hope to gain. The following product deelopment phases can especially benefit from receiving feedback from Product Market Fit surveys.
- Slow Business Growth. If your business is experiencing slow growth, conducting a survey can provide valuable insights into the interests and needs of your target market.
- Scaling and Fundraising. Before acquiring funding to scale your business, it is essential to gather insights into your customers’ interests and needs.
- Product or Feature Changes. If your product has undergone a significant change or a new feature has been introduced, surveying customers can provide insight into how they perceive and interact with the new offering.
- A/B Testing and Ad-hoc Surveys. Conducting ad-hoc surveys during the A/B testing phase or for a momentary marketing change can provide maximum output and valuable insights.
Ongoing Product market fit surveys
As you progress on your journey toward creating a successful product, the Product Market fit survey result goes from being a key result you are chasing to being another health metric or KPI that you are continuously monitoring. Just like the Net Promoter Score (NPS). In this case, you want to contiously ask a sample of customers or users from various segments and in various stages of your product life-cycle. Try to avoid asking the same person twice in in the middle of the lifecycle, but insist on for instance including the survey question in your exit-survey.
This will allow you to monitor how customer sentiment changes over time as users progress from being just onboarded, to being active, to existing.
Product Market Fit Surveys grows in importance as your product grows
As a product grows, it can be challenging to determine whether new features or offerings are truly valuable to customers. It’s crucial to maintain a laser focus on key metrics, even after achieving product market fit.
To avoid getting carried away by positive buzz, strive to separate marketing and product launch as much as possible, keeping new features or products in beta until they are confident they provide value to customers.
This is where the Product Market Fit survey becomes especially valuable. By conducting surveys before the general public is aware of a new feature or product, you can gather valuable feedback and determine the feature or product’s effectiveness.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Popular tools
The tools below will help you with the Product-Market Fit Survey play.
-
PMF Survey
A free tool by Sean Ellis to measure Product-Market fit as defined by the man himself.
Examples
Survey on Slack
An open research project with the mission of discovering why Slack is so addictive found that it not only hard Product-Market fit, but also why it worked annd what could be improved.
Buffer
Buffer determined that 40-50 responses should carry statistical significance
Source: A Simple Guide to Measuring the Product-Market Fit of Your Product or Feature
Related plays
- How to find Product/Market fit by Ryan Law
- False Positives and Product Market Fit by Tristan Kromer
- Product/Market Fit Survey by Tristan Kromer
- Part 4: The only thing that matters by Marc Andreessen
- A Simple Guide to Measuring the Product-Market Fit of Your Product or Feature by Leo Widrich
- The Real Startup book - Product/Market Fit Survey by Tristan Kromer, et. al.
- Product Market Fit Survey: Best Practices and Examples by Ramnish
- Sean Ellis Score by Anders Toxboe