The Wizard of Oz Experiment Has a New Wizard: AI

Posted by Anders Toxboe on October 16, 2026 · 12 mins read

---

The Wizard of Oz experiment has always been one of my favorite Lean Startup techniques.

The idea is simple: Instead of building an expensive product, you manually deliver the value it promises. From the customer’s perspective, it looks like a functioning product. Behind the scenes, a human does the work.

The founders of Zappos tested whether people would buy shoes online without first stocking an inventory. They went to local shoe stores, photographed the shoes on the shelves, and listed them online. When someone placed an order, they returned to the store, bought the pair at retail price, and shipped it to the customer.

The experiment simulated an online shoe retailer before the business had its own inventory and fulfillment operation. The founders weren’t just manually shipping shoes. They were testing whether customers would buy footwear online at all.

It was inefficient, deliberately unscalable, and a brilliant way to validate demand.

But something interesting is happening to this experiment.

The human behind the curtain can now be replaced by AI.

Suddenly, what used to work for 10 customers might work for 100 or even 1,000.

That changes the economics of experimentation. But it also introduces some interesting paradoxes about how we learn.

A Wizard of Oz experiment that scales

I recently experimented with this approach at Learning Loop.

I created a free Persuasion Review tool where you can submit a website and receive a personalized review based on our 80 Persuasive Patterns.

You provide a URL, specify what you want visitors to do, and receive an analysis of where people might get stuck, the psychology behind those problems, and what to improve first.

Traditionally, delivering this kind of analysis would require a UX consultant to manually inspect every website and prepare a report.

Instead, AI can perform much of that analysis.

This allows me to experiment with a personalized service without first building a sophisticated production system.

It might require expensive models, large context windows, multiple AI calls, and quality checks. The cost per report might be much higher than I’d accept in a mature product.

But that’s not necessarily a problem.

Suppose an AI review costs $5 to deliver. A thousand requests would mean $5,000 in inference costs alone—not including development, quality checks, support, or acquisition. That may still be a reasonable price to learn whether the service is valuable before investing months in a production system.

The goal of an experiment isn’t to deliver the service efficiently. It’s to learn whether the service is worth building.

AI makes it possible to postpone many engineering decisions while testing demand at a scale that previously required substantial manual labor.

Strictly speaking, once AI is genuinely performing the work, this isn’t a traditional Wizard of Oz experiment anymore. It’s an AI-powered variation in which the fulfillment is partially automated, but the complete product hasn’t been built.

And while the opportunity is exciting, I think it introduces three important paradoxes.

Paradox #1: More customers, less empathy

One of the overlooked benefits of traditional Wizard of Oz experiments is that the founder has to do the work.

When you personally fulfill customer requests, you encounter all kinds of situations you wouldn’t otherwise discover.

You see the unusual requests. You notice where customers struggle to explain their problems. You begin recognizing similarities between seemingly unrelated needs.

You learn not just what customers want, but why.

Over time, these experiences help develop product sense and taste. You become better at judging which problems matter, which solutions feel right, and which details make a meaningful difference.

That understanding eventually helps shape your product vision.

But what happens when AI performs the work?

You might successfully serve 1,000 customers without personally engaging with a single one.

You’ll have more customers, more interactions, and potentially more feedback than ever before. Yet you might develop less intuition about the people you’re serving.

The risk isn’t just losing customer feedback. It’s losing the experiences that teach you how to interpret it.

For my Persuasion Review experiment, this means I shouldn’t just count how many reviews are requested. I should personally inspect some of the websites, review the AI’s recommendations, and talk to people about what they found valuable.

A scalable experiment still needs unscalable moments of human learning.

Paradox #2: More qualitative data, but weaker evidence?

Traditional Wizard of Oz experiments can generate both evaluative evidence about whether a solution works and generative insights about customer needs.

At a small scale, you can read customer responses, examine individual behaviors, and understand the context behind them.

But imagine receiving thousands of requests, conversations, and feedback responses.

You probably won’t read all of them yourself.

Instead, you’ll use AI to summarize responses, identify recurring themes, cluster problems, and quantify patterns.

Suddenly, your qualitative research starts to look like quantitative research.

An AI summary might report that 38% of customers mentioned one problem and 22% requested a particular capability.

These numbers feel authoritative. But we need to be careful.

Aggregating opinions doesn’t turn them into facts.

AI can classify thousands of statements, but the underlying evidence is still what people say.

Customers are often unreliable predictors of their own future behavior. They may enthusiastically describe a feature they’d never use or praise a service they’d never pay for.

And AI-generated analysis adds another layer of interpretation that can introduce bias or false precision.

This is why I believe the most useful AI-powered experiments should collect behavioral evidence wherever possible.

Did customers actually request a review? Did they read it? Did they implement any recommendations? Did they return? Did they recommend it to someone else? Would they pay?

These actions generally provide stronger evidence of actual demand than a positive response to a feedback question, although even a free request is a weaker signal than paying.

The challenge is to combine AI’s ability to analyze large amounts of qualitative data with experiments designed to capture what customers actually do.

Paradox #3: You might discover a different product

There’s another interesting consequence of letting AI deliver the experience.

Traditional Wizard of Oz experiments usually simulate a relatively well-defined product. You have a hypothesis about what customers need and create an experience to test it.

But AI-powered services can be surprisingly flexible.

You might start with one intended use case and discover that customers are trying to accomplish something entirely different.

Imagine that people using my Persuasion Review tool turn out to be less interested in the comprehensive website analysis than in generating specific redesign suggestions.

Or perhaps they value competitor comparisons more than the persuasion review itself.

Maybe what they really want is help deciding which conversion experiments to run next.

These are hypothetical examples, but they illustrate an important opportunity.

You might set out to validate one product and discover demand for another.

Instead of only measuring whether customers use the service as intended, you can look for patterns in the jobs they’re trying to get done.

This requires more than counting feature requests.

Two customers might ask for completely different outputs while trying to accomplish the same underlying job. Conversely, two customers requesting the same feature might have entirely different motivations.

AI can help identify these patterns across many interactions.

But the product team still needs to interpret them, investigate the underlying needs, and decide which opportunities deserve attention.

The experiment becomes more than a way to validate a predefined solution. It becomes a way to discover what the solution should be.

How to scale the experiment without losing the learning

So how do we take advantage of AI-powered Wizard of Oz experiments without sacrificing what made them valuable in the first place?

I think a few principles are worth following:

1. Stay involved in fulfillment. Personally handle some requests, inspect AI-generated outputs, and investigate unusual cases. Don’t automate yourself entirely out of the learning process.
2. Keep talking to customers. AI-generated reports and summaries are useful, but they aren’t substitutes for firsthand conversations and observations.
3. Design for behavioral evidence. Measure what people actually do, not just what they say. Repeat usage, referrals, implementation, and willingness to pay can tell you more than satisfaction scores alone.
4. Treat AI analysis as hypothesis generation. Use AI to discover potential patterns, then validate important findings against real interactions and behavior.
5. Look for unexpected jobs to be done. Don’t become so focused on validating your original idea that you overlook what customers are actually trying to accomplish.

The important distinction is between automating delivery and outsourcing the judgment that turns observations into learning.

Beyond Wizard of Oz: Other experiments face similar paradoxes

These trade-offs aren’t unique to Wizard of Oz experiments. AI is changing several other product validation experiments.

In a Concierge experiment, AI can help deliver personalized services to many more customers, but risks removing the direct interactions that help founders develop empathy and product intuition.

In a Mashup experiment, AI can combine existing services into sophisticated experiences without much engineering. But validating customer demand becomes easier than validating whether the experience can be delivered reliably and profitably.

And in a Single-Feature Product experiment, even a narrowly scoped AI capability can accommodate unexpected use cases. This creates opportunities to discover new jobs to be done, but makes it harder to know which product hypothesis has actually been validated.

Across these experiments, the same question keeps appearing: Are we using AI to learn more about our customers, or simply automating the activities through which we used to learn?

Scale delivery, not distance from customers

AI removes one of the biggest limitations of traditional Wizard of Oz experiments: the human effort required to serve each additional customer.

But that human effort was never entirely wasteful. It was also how founders developed empathy, discovered unexpected needs, and built the judgment necessary to create good products.

The opportunity isn’t simply to run experiments at a much larger scale.

It’s to learn from more customers without losing the understanding we used to gain from serving them one at a time.