Idea Validation: Product

Multivariate Testing

Test which combination of components performs better

Illustration of Multivariate Testing
Run a Multivariate Testing play

Also called: Full Factorial Testing

Difficulty: Intermediate

Requires existing audience or product

Evidence strength
38

Relevant metrics: Unique views, Conversions

Validates: Desirability

How: Determine which combination of variations performs the best out of all of the possible combinations. Where an A/B test only tests one variable, multivariate testing tests multiple variables together to uncover the ideal combination that is effective in improving the ultimate goal.

Why: By changing multiple elements on the same webpage in tandem to improve a single conversion goal, multivariate testing can eliminate the need to run several sequential A/B tests on the same page with the same goal. However, it requires significantly more traffic (exposure) than A/B testing as combinations to test increase exponentially with each added component.

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

Similar to A/B testing, but more detailed

Once your website has reached a traffic level to allow multi-variate testing, you can step up from not only comparing two or more variations of an element, but how elements work together in harmony. Multivariate testing can potentially reveal what elements have the greatest positive or negative impact on the interactions of your users.

Duration is exponentially connected to number of variations

However, multivariate testing comes with one caveat: the more variations that you test, the longer it takes to obtain meaningful data from the test. This is why multivariate testing is often recommended only for sites with a substantial amount of daily traffic - not only as a whole, but also on individual pages.

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Popular tools

The tools below will help you with the Multivariate Testing play.

  • Optimizely

    Enterprise A/B testing tool including personalisation and recommendation modules for more advanced optimisation

  • Visual Website Optimizer (VWO)

    Established A/B testing suite including testing & experimentation, research & user feedback, analytics & reporting, and targeting & personalization..

  • Adobe Target

    Similar features to other enterprise testing tools such as Optimizely, including personalisation and recommendation modules for more advanced optimisation

Examples

Amazon testing platform

Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.

Source: A/B Testing at Scale – Amazon Machine Learning Research

Sources

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Set goals

Convert empathy to clarity by refining insights into problem definitions

Ice Breakers

Relieve initial group awkwardness and establish a safe space

Community events
Product Loop

Product Loop provides an opportunity for Product professionals and their peers to exchange ideas and experiences about Product Design, Development and Management, Business Modelling, Metrics, User Experience and all the other things that get us excited.

Join our community

Made with in Copenhagen, Denmark

Want to learn more about about good product development, then browse our product playbooks.