Also called: Product Reviews
See also: Customer Service Logs, Industry Forums
Difficulty: Intermediate
Evidence strength
Relevant metrics: Jobs to be done ranking, Ranking needs, wants, desires, pains, Quotes & stories
Validates: Desirability
How: Read reviews of your own and competing products in app stores and on review sites and distill emotional outbursts, frustrations, and requests to tangible insights. Consider whether users are missing features, experiencing confusing UI, complaining about bugs or performance, or could need better onboarding.
Why: The frustrations and pain points of your own and competing products are shared without filter and in all honesty in app reviews. App reviews provide free access to feedback from existing and potential customers.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
Why are App reviews interesting?
App reviews are not just ratings; they are a direct line of communication from your users. They offer unfiltered insights into what users like, dislike, and wish to see in your app. By analyzing these reviews, you gain a deeper understanding of user needs, preferences, and pain points.
Don’t mind the rating
When looking for insights on your own product or competing products on the App Store or Google Play, you should focus on the substance of what is being said rather than the rating itself (typically one to five stars).
You could look for these queus:
- Are users annoyed?
- Are users asking for specific features to be added?
- Are users struggling with specific parts of the experience
- Are uses excited about the product or part of the experience?
- Are users complaining about bugs or performance issues?
- Are users loving features not indended as part of the main experience (that should consider promoting)?
- Do users understand how to get started, or would it seem appropriate to add a tutorial?
- Are there any parts of the user experience that doesn’t get mentioned?
- Are users just loyal to the brand above anything else?
Reviews tend to be emtionally driven outliers
Users are most likely to leave feedback as they are motivated by an emotional reaction to the product. Emotional triggers, positive or negative, tend to happen by outliers on the bell curve. This is why it’s important to spend time distilling your findings into tangible insights.
Research your competitors
Reading app reviews is a great chance to study competing products – not only your own. Can you spot a frustation or feature gap in a competitors app that you can close?
Reaching out to reviewers
A valueable way to extend this expriment is to directly reach out to users who have reviewed your app. This can be done by finding their Apple ID from the review post or by interacting with followers on the app’s social media pages. Such direct contact with users provides invaluable insights into what users actually want from the app, which can then be used to refine the product.
Strategies for reaching out to reviewers include:
- Identify valuable reviews. Start by identifying reviews that offer constructive feedback. Look for detailed comments that discuss specific features, usability issues, or enhancement suggestions.
- Establish a contact method. For users who have left reviews on platforms like the Apple App Store, it’s possible to find their contact details like Apple ID. Alternatively, you can engage with users on your app’s social media pages, as users often use these platforms to express their opinions.
- Draft a response template. Prepare a template for reaching out to users. Your message should be polite, appreciative of their feedback, and express your interest in further understanding their experience. Personalize each message to reflect the specific feedback the user has provided.
- Engage in a dialogue. Encourage an open dialogue. Ask specific questions about their experience and suggestions. Show that you are genuinely interested in their input and that their feedback is valuable to the development of the app.
- Document and analyze feedback. Keep a record of all interactions and feedback received. Analyze this data to identify common themes or recurring issues. This information is crucial for prioritizing development efforts.
- Follow-up. Once changes are made based on user feedback, follow up with the users who provided that feedback. Inform them about the updates and express gratitude for their contribution. This not only validates their effort but also fosters a sense of community and loyalty.
By listening to and acting upon user feedback, you demonstrate a commitment to continuous improvement, which is essential in the dynamic field of app development.
In the end, your goal is to create a product that resonates with your users, and there is no better way to achieve this than by directly understanding their needs and experiences.
Using semantic aalysis to analyze reviews at scale
Semantic analysis, at its core, involves the interpretation and classification of words and phrases within the app reviews to discern their underlying meanings and sentiments. This process transcends mere word counting or simplistic analysis; it delves into the context and connotations of language used by the app users. By applying natural language processing techniques, semantic analysis can categorize reviews into sentiments like positive, negative, or neutral.
The value of this approach lies in its ability to process large volumes of text data - far more than what a human team could feasibly analyze manually. It systematically identifies patterns and trends in user feedback, which might otherwise be overlooked. For example, an increase in negative sentiment in reviews after an app update can quickly signal a problem that needs immediate attention, such as a new bug or an unpopular feature change.
Moreover, semantic analysis helps in pinpointing specific aspects of the app that users frequently discuss. It can highlight features that are particularly appreciated or aspects that consistently cause frustration. This data-driven approach allows product managers to prioritize their development efforts based on what matters most to their users.
Identifying feature requests through semantic analysis is another significant advantage. Users often suggest improvements or new features in their reviews. Semantic analysis can aggregate these suggestions and identify the most commonly requested features, providing a direct line of insight into user needs and expectations.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Popular tools
The tools below will help you with the Read App Reviews play.
-
AppFollow
Utilizing machine learning for semantic analysis of user reviews, this tool provides deep insights into customer satisfaction and pain points. This approach can enable rapid, data-driven adjustments to the product roadmap.
-
Appbot
Appbot provides review & ratings monitoring, replies and analysis.
Examples
Recurly
For Recurly, the customer support team is the fastest and easiest way to get feedback from customers. By talking with the team, listening in on calls, and working to identify beta customers, customer support is an integrated part of product development.
Source: The UX of Customer Support
PicsArt
PicsArt utilized semantic analysis on user reviews to gauge sentiment and gather feedback. This approach automated product adjustments and identified popular feature requests, aiding in effective and user-focused product development.
Source: Semantic analysis: How to understand user reviews at scale and drive customer satisfaction
Related plays
- User Research When You Can’t Talk to Your Users by Jon Peterson
- The UX of Customer Support by Recurly
- How to Use App Reviews for Market Research by Carter Thomas