Also called: Full Factorial Testing
See also: A/B Testing, Conjoint Analysis
Difficulty: Intermediate
Requires existing audience or product
Evidence strength
Relevant metrics: Unique views, Conversions
Validates: Desirability
How: Determine which combination of variations performs the best out of all of the possible combinations. Where an A/B test only tests one variable, multivariate testing tests multiple variables together to uncover the ideal combination that is effective in improving the ultimate goal.
Why: By changing multiple elements on the same webpage in tandem to improve a single conversion goal, multivariate testing can eliminate the need to run several sequential A/B tests on the same page with the same goal. However, it requires significantly more traffic (exposure) than A/B testing as combinations to test increase exponentially with each added component.
This experiment is part of the Validation Patterns printed card deck
A collection of 60 product experiments that will validate your idea in a matter of days, not months. They are regularly used by product builders at companies like Google, Facebook, Dropbox, and Amazon.
Get your deck!Before the experiment
The first thing to do when planning any kind of test or experiment, is to figure out what you want to test. To make critical assumptions explicit, fill out an experiment sheet as you prepare your test. We created a sample sheet for you to get started. Download the Experiment Sheet.
Similar to A/B testing, but more detailed
Once your website has reached a traffic level to allow multi-variate testing, you can step up from not only comparing two or more variations of an element, but how elements work together in harmony. Multivariate testing can potentially reveal what elements have the greatest positive or negative impact on the interactions of your users.
Duration is exponentially connected to number of variations
However, multivariate testing comes with one caveat: the more variations that you test, the longer it takes to obtain meaningful data from the test. This is why multivariate testing is often recommended only for sites with a substantial amount of daily traffic - not only as a whole, but also on individual pages.
After the experiment
To make sure you move forward, it is a good idea to systematically record your the insights you learned and what actions or decisions follow. We created a sample Learning Sheet, that will help you capture insights in the process of turning your product ideas successful. Download the Learning Sheet.
Popular tools
The tools below will help you with the Multivariate Testing play.
-
Optimizely
Enterprise A/B testing tool including personalisation and recommendation modules for more advanced optimisation
-
Visual Website Optimizer (VWO)
Established A/B testing suite including testing & experimentation, research & user feedback, analytics & reporting, and targeting & personalization..
-
Adobe Target
Similar features to other enterprise testing tools such as Optimizely, including personalisation and recommendation modules for more advanced optimisation
Examples
Amazon testing platform
Using their homegrown real time bandit algorithm, Amazon’s multivariate testing platform can find the best performing combination of 48 page elements seven times faster than predicted for traditional A/B testing.
Source: A/B Testing at Scale – Amazon Machine Learning Research
Related plays
- Multivariate Testing vs A/B Testing by Optimizely