Also called: Full Factorial Testing
See also: A/B Testing, Conjoint Analysis
Difficulty: Intermediate
Requires existing audience or product
Evidence strength
Relevant metrics: Unique views, Conversions
Validates: Desirability
How: Determine which combination of variations performs the best out of all of the possible combinations. Where an A/B test only tests one variable, multivariate testing tests multiple variables together to uncover the ideal combination that is effective in improving the ultimate goal.
Why: By changing multiple elements on the same webpage in tandem to improve a single conversion goal, multivariate testing can eliminate the need to run several sequential A/B tests on the same page with the same goal. However, it requires significantly more traffic (exposure) than A/B testing as combinations to test increase exponentially with each added component.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
Similar to A/B testing, but more detailed
Unlike A/B testing, which compares two versions of a single variable, multivariate testing examines the impact of various combinations of changes across multiple elements.
Once your website has reached a traffic level to allow multi-variate testing, you can step up from not only comparing two or more variations of an element, but how elements work together in harmony. Multivariate testing can potentially reveal what elements have the greatest positive or negative impact on the interactions of your users.
Duration is exponentially connected to number of variations
However, multivariate testing comes with one caveat: the more variations that you test, the longer it takes to obtain meaningful data from the test. This is why multivariate testing is often recommended only for sites with a substantial amount of daily traffic - not only as a whole, but also on individual pages.
Depth of insights and evidence strength
An important benefit of multivariate testing is the depth of insights it provides. It helps in understanding not just the effectiveness of individual changes but also the synergy between different elements of the product. For instance, a change in the user interface might be more effective when combined with a specific feature enhancement, an insight that might be missed in simpler forms of testing.
However, multivariate testing can be complex and requires a significant amount of traffic or user interactions to achieve statistical significance. It also demands a robust analytical framework to correctly interpret the interactions between multiple variables. Despite these challenges, the insights gained from multivariate testing can be invaluable in creating a product that resonates well with the target audience and stands out in the market.
It is a sophisticated tool in the product manager’s arsenal for product validation. It enables a deeper understanding of the product and its interaction with users, providing a comprehensive view of how various elements work together. This approach is graet for fine-tuning products and their intricate tiny details, leading to more informed decisions and ultimately, a better user experience and success of the product.
Why use multivariate testing for product reserach?
The method is designed to evaluate the performance of different combinations of website or product components.
Multivariate testing aims to identify the most effective combination of multiple variables that influence user behavior and conversion rates. By simultaneously testing variations of several elements, businesses can gain insights into how these variables interact with each other and their collective impact on the user experience and conversion goals. This approach is more complex and data-intensive than A/B testing but offers comprehensive insights that can lead to more significant improvements in performance.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Popular tools
The tools below will help you with the Multivariate Testing play.
-
Optimizely
Enterprise A/B testing tool including personalisation and recommendation modules for more advanced optimisation
-
Visual Website Optimizer (VWO)
Established A/B testing suite including testing & experimentation, research & user feedback, analytics & reporting, and targeting & personalization..
-
Adobe Target
Similar features to other enterprise testing tools such as Optimizely, including personalisation and recommendation modules for more advanced optimisation
Examples
Amazon testing platform
Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.
Source: A/B Testing at Scale – Amazon Machine Learning Research
Jawbone
Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.
Bob’s Family Store
A small-town retail shop, Bob’s Family Store, which also runs an online store, used multivariate testing to optimize its marketing efforts around holidays and weekends. The store utilized multivariate testing to determine the most effective marketing messages and strategies for different times of the year, such as Black Friday and other major shopping days.
Related plays
- Multivariate Testing vs A/B Testing by Optimizely