In product research, especially as advocated by continuous product discovery, facts take precedence over opinions. When you ask for opinions, you get opinions. People have opinions on everything, and they most likely do not reflect how people actually behave in the real world. This is why there is a clear distinction between gathering opinions and uncovering real-world user behaviors.
Focus groups have traditionally served as a method for collecting opinions and sentiments from potential users, offering insights into how customers perceive messaging, branding, and overall positioning. However, focus groups often fall short as a reliable method for identifying genuine user problems or verifying market demand. They can tell you what people say they want, but they rarely confirm what people will actually do.
When AI is used as a replacement for talking to real users and seeking real evidence - it is much like a focus group – it can be quite effective in evaluating messaging and sentiment. When used as a tool to interpret real evidence, AI tools, like large language models, can be a life-saver. It can summarize user feedback and perform complex calculations that integrate multiple datasets.
But used alone, it’s better for refining how a product or service is communicated. For tasks such as testing headlines, clarifying product descriptions, or assessing initial comprehension, AI can quickly surface insights and patterns, effectively simulating the opinion-based input one might expect from a traditional focus group. If the goal is to refine your product’s messaging and assess overall reception, AI-driven analysis aligns well with the strengths of focus groups.
However, the moment the goal shifts from gathering opinions to uncovering factual evidence of what users do, AI should not be the primary source of research. Good product discovery focuses on uncovering real customer problems, not just what participants claim they might do in a hypothetical scenario. Asking questions like “Would you…” or “Do you think…” often leads to unsubstantiated opinions. People will say what they believe you want to hear, even if it does not match their actual behavior.
AI should not be the primary source of research
Instead, product discovery should rely on methods that capture real actions and genuine struggles. Asking “When was the last time you…” prompts concrete, fact-based responses. These kinds of questions aim to surface instances where a user actually experienced a problem. For example, “When was the last time you tried to plan your meals for the week?” or ““When was the last time you needed to find a quick recipe for dinner?” forces the respondent to recall a real event, rather than speculate on a future action. It prompts further questioning and investigation about the context of the instance. Real-world details — such as when users last researched a solution, what triggered that need, and how they addressed it — are the building blocks of evidence-based discovery.
By focusing on past behaviors, product teams can understand the actual pain points and priorities that drive user decisions. This approach allows product managers and user researchers to distinguish between what customers claim they would do and what they truly have done. These insights help determine the real value proposition of a product and validate whether customers are willing to invest time, energy, or money to solve a particular problem.
In other words, AI is great as a tool to make sense of real data. Used as the primary source of input, AI is more well-suited for accelerating and refining the collection of opinions — much like a virtual focus group. It’s an echo chamber of what has already been said. AI excels at synthesizing existing data available to Large Language Models (LLMs). However, it can never substitute experiments or interviews that anchor insights in reality. Product discovery demands evidence-based methods that are designed to uncover genuine user needs, validate market size, confirm whether the product solves the identified problem, and assess if users are ready to pay for it.
When you need to perfect how you talk about a product, AI can be a valuable asset, similar to a well-run focus group. However, if your goal is to engage in true product discovery — identifying what users really do and what they truly need — AI alone will not suffice. Instead, rely on techniques that draw out factual stories and verifiable user behaviors. This ensures that product decisions are based on solid evidence, rather than solely on what users claim they might want.
When product teams use AI (or any tool), its vital they understand its limitations and strengths. AI can refine and simulate, but it can’t replace the hard-won insights that come from observing and interacting with real users. My case is not rejecting AI as a tool, but about using the right tools for the right job.
- The Mom Test by Rob Fitzpatrick
- Continuous Discovery Explained by Teresa Torres
