Last week I read in a report from one of the big strategic consultancy firms: “you should be doing multivariate testing”.
This advice seems logical. Multivariate testing is a way to test a group of elements on a page. In theory, it’s a great way to see the impact of several changes to the UI, design and copy. And it gives insight in how these elements play together.
When I asked my LinkedIn network full of A/B testing and CRO experts:
“Should your experimentation tool have the ability to do multivariate testing?”
The overwhelming answer was: YES (70% from 157 participants).
But …
When I asked this follow-up question:
“Of the last 10 experiments you ran, how many were multivariate tests (MVT)?”
The respons completely flipped.
Now 77% says they have focussed on pure A/B/n testing. 17% said they did one, maybe two MVT experiments. Only 5% did more than three.
As a former Optimizely employee confirmed:
“As someone who has worked for Experimentation platform vendors, run platform evaluations for agencies, and runs tests themselves, it amazes me the volume of folks who demand ‘MVT’ during their purchase and then either don’t use it at all OR can’t explain why it’s important to them. It’s as if having it, is more important than using it. There were only a tiny proportion of Optimizely customers that actually ever used MVT, and an even smaller group successfully.”
Simon Herron
The reality is that multivariate testing is easy to sell, but hard to actually do. And most companies lack the traffic volume needed for proper multivariate testing.
So before you start asking for multivariate testing capabilities, make sure you know why you want it, check if you have the proper amount of traffic and make sure you have reached a mature level of doing A/B/n testing first.
