Idea Validation: Market demand, Product

Wizard of Oz

Use human power to fake automation of complex tasks

Illustration of Wizard of Oz
Run a Wizard of Oz play

Also called: Manual-first, Mechanical Turk

See also: Concierge

Difficulty: Intermediate

Evidence strength
90

Relevant metrics: Customer satisfaction, Activation count, Cycle time, Purchase coun

Validates: Feasibility, Viability, Desirability

How: Use your own hands, an intern, or online crowdsourcing services to fake automation of tasks that are now too costly to build. To keep the facade of no human involvement, consider constructing your experiment so that real-time response is not needed in order to deliver on the value proposition you're testing.

Why: Humans can be cheaper than automation. Even if this takes longer for the customer to receive an answer, you will avoid wasting precious time building features the customer does not want.

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

Creating an illusion to prope demand

Using the Wizard of Oz experiment, you are creating an illusion of an actual intended product, where the unsuspecting user will never know that you are in fact handling all suposedly automated tasks, manually. From the point of view of the customer, your Wizard of Oz prototype will do the job in a way that end-users are not aware that humans are working behind the scenes.

Why would you do this?

First, a Wizard of Oz prototype provides a unique opportunity to verify the demand of your product that, so far, only exists in your mind. So why spend time and money building an elaborate system to handle and automate your customers’ requests, when you can just test whether anybody are interested at all in your future product, by hand-holding the entire process yourself.

Secondly, conducting all tasks manually will provide unique insights into what it takes to deliver customer value as well as how users react when you finally deliver value to them.

A flexible solution to adjust as you learn

Wizard of Oz testing is a fast and effective way to test your hypothesis as to whether your proposed solution will in fact create value for your potential customers. At the same time, performing all tasks manually comes with the added benefit of being easily adjustable. Being able to quickly modify your hand-held product experience lets you test a larger number of hypotheses, quickly – finding the most effective solution, fast.

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Popular tools

The tools below will help you with the Wizard of Oz play.

  • Mechanical Turk

    This Amazon service provides access to a global, on-demand 24/7 workforce conducting simple tasks called HIT – "Human Intelligence Tasks".

  • MicroWorkers

    An alternative to the original Mechanical Turk service by Amazon that provide a series of ready-to-use templates.

Examples

Zappos

Zappos founder, Nick Swinmurn, decided to test the assumption that people would be willing to buy shoes online without trying them on. Instead of building an inventory and then trying to sell it, Nick took a different approach. He went to local stores, photographed shoes, and then advertised them online. If a pair of shoes were sold, Nick would go back to the store, buy the shoe, and send it to the customer.

From the customers perspective, it would look like Zappos had a full inventory of shoes, while in fact, all orders were processed in this unscalable way manually. Nick could use this experiment to fine tune marketing, wording, packeting, and product categories in terms of what constituted better product/market fit. Once momentum was gained, Zappos could push play and invest in expensive inventory and handling as the certainty of success was maximized.

Source: The Wizard of Oz MVP

Aadvark

The Q&A service, Aadvark, routes questions to expert users (via instant messaging). In the early days, the Aadvark staff would manually post the questions to whomever was online to see if that particular user would respond, and then manually post the answer back to the asker. There was no automation or algorithm. The feature was later built to automate this functionality. From the outside, this MVP looked like a fully functional system, even though most tasks were manually executed behind the scenes by a human.

Source: Discover the 4 types of Minimum Viable Product

Testing speech recognition

To test the usefulness of an app involving speech recognition without actually implementing speech recognition, a usability test was set up where the participant talked in a microphone and a typist listening next door made the right words appear on the participant’s screen.

IBM Speech Recognition

IBM tested market interest in speech-to-text technology by using a dummy computer and a hidden typist to simulate a functioning prototype. This experiment provided valuable insights into user acceptance and feasibility of the technology.

Source: Pretotyping – Pretending to Prototype

Related plays

Sources

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Ice Breakers

Relieve initial group awkwardness and establish a safe space

Community events
Product Loop

Product Loop provides an opportunity for Product professionals and their peers to exchange ideas and experiences about Product Design, Development and Management, Business Modelling, Metrics, User Experience and all the other things that get us excited.

Join our community

Made with in Copenhagen, Denmark

Want to learn more about about good product development, then browse our product playbooks.