Idea Validation: Product

Multivariate Testing

Test which combination of components performs better

Illustration of Multivariate Testing
Run a Multivariate Testing play

Also called: Full Factorial Testing

Difficulty: Intermediate

Requires existing audience or product

Evidence strength
38

Relevant metrics: Unique views, Conversions

Validates: Desirability

How: Determine which combination of variations performs the best out of all of the possible combinations. Where an A/B test only tests one variable, multivariate testing tests multiple variables together to uncover the ideal combination that is effective in improving the ultimate goal.

Why: By changing multiple elements on the same webpage in tandem to improve a single conversion goal, multivariate testing can eliminate the need to run several sequential A/B tests on the same page with the same goal. However, it requires significantly more traffic (exposure) than A/B testing as combinations to test increase exponentially with each added component.

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!

Before the experiment

The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.

Similar to A/B testing, but more detailed

Unlike A/B testing, which compares two versions of a single variable, multivariate testing examines the impact of various combinations of changes across multiple elements.

Once your website has reached a traffic level to allow multi-variate testing, you can step up from not only comparing two or more variations of an element, but how elements work together in harmony. Multivariate testing can potentially reveal what elements have the greatest positive or negative impact on the interactions of your users.

Duration is exponentially connected to number of variations

However, multivariate testing comes with one caveat: the more variations that you test, the longer it takes to obtain meaningful data from the test. This is why multivariate testing is often recommended only for sites with a substantial amount of daily traffic - not only as a whole, but also on individual pages.

Depth of insights and evidence strength

An important benefit of multivariate testing is the depth of insights it provides. It helps in understanding not just the effectiveness of individual changes but also the synergy between different elements of the product. For instance, a change in the user interface might be more effective when combined with a specific feature enhancement, an insight that might be missed in simpler forms of testing.

However, multivariate testing can be complex and requires a significant amount of traffic or user interactions to achieve statistical significance. It also demands a robust analytical framework to correctly interpret the interactions between multiple variables. Despite these challenges, the insights gained from multivariate testing can be invaluable in creating a product that resonates well with the target audience and stands out in the market.

It is a sophisticated tool in the product manager’s arsenal for product validation. It enables a deeper understanding of the product and its interaction with users, providing a comprehensive view of how various elements work together. This approach is graet for fine-tuning products and their intricate tiny details, leading to more informed decisions and ultimately, a better user experience and success of the product.

Why use multivariate testing for product reserach?

The method is designed to evaluate the performance of different combinations of website or product components.

Multivariate testing aims to identify the most effective combination of multiple variables that influence user behavior and conversion rates. By simultaneously testing variations of several elements, businesses can gain insights into how these variables interact with each other and their collective impact on the user experience and conversion goals. This approach is more complex and data-intensive than A/B testing but offers comprehensive insights that can lead to more significant improvements in performance.

After the experiment

To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.

Popular tools

The tools below will help you with the Multivariate Testing play.

  • Optimizely

    Enterprise A/B testing tool including personalisation and recommendation modules for more advanced optimisation

  • Visual Website Optimizer (VWO)

    Established A/B testing suite including testing & experimentation, research & user feedback, analytics & reporting, and targeting & personalization..

  • Adobe Target

    Similar features to other enterprise testing tools such as Optimizely, including personalisation and recommendation modules for more advanced optimisation

Examples

Amazon testing platform

Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.

Source: A/B Testing at Scale – Amazon Machine Learning Research

Jawbone

Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.

Source: Examples of Multivariate Testing in Marketing

Bob’s Family Store

A small-town retail shop, Bob’s Family Store, which also runs an online store, used multivariate testing to optimize its marketing efforts around holidays and weekends. The store utilized multivariate testing to determine the most effective marketing messages and strategies for different times of the year, such as Black Friday and other major shopping days.

Source: Examples of Multivariate Testing in Marketing

This experiment is part of the Validation Patterns printed card deck

A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.

Get your deck!
Sources

Want to learn more?

Receive a hand picked list of the best reads on building products that matter every week. Curated by Anders Toxboe. Published every Tuesday.

No spam! Unsubscribe with a single click at any time.

Ice Breakers

Relieve initial group awkwardness and establish a safe space

Educate

Broaden knowledge or insight regarding the behavior or situation to inform decisions.

Demonstrate

Show practical examples or models of the desired behavior for clear guidance.

Alert

Highlight current actions and their reasons, bringing unconscious habits to awareness.

Train

Develop necessary skills and competencies to enable effective action.

Community events
Product Loop

Product Loop provides an opportunity for Product professionals and their peers to exchange ideas and experiences about Product Design, Development and Management, Business Modelling, Metrics, User Experience and all the other things that get us excited.

Join our community

Made with in Copenhagen, Denmark

Want to learn more about about good product development, then browse our product playbooks.