Abridged with the author’s permission from the MarketingExperiments blog post entitled Marketing Optimization: How to design split tests and multi-factorial tests. This is a follow-up to a post we abridged from Daniel Burstein a month ago on how to formulate a meaningful marketing research question.
In Daniel Burstein’s blog post on research questions, we emphasized the importance of asking “which” rather than “what” questions because a “which” question is clearly testable. For example, “Which page format results in the most lead submissions?” The question is clearly stated and includes two key pieces of information:
- An independent variable you are going to test
- The dependent variable you will use to measure your results
To know if something is better, first you must know if it is different
With the research question on paper, we can easily create a hypothesis. For the former question: “All page formats will result in the same number of lead submissions.” This type of hypothesis is so famous in research circles that it has a name: “The Null Hypothesis.”
In general terms, the null hypothesis states that varying the independent variable (the page) will result in no change to the dependent variable (the number of leads).
Control vs. Treatment(s)
In most cases, there will be an existing page that all new versions will be compared to. This page is termed the “Control,” and all new pages are dubbed “Treatments” to guide comparisons later.
The next step in testing your research question is to decide on the most appropriate test structure. This will depend on the number of variations you will be testing, and on the amount of traffic your site receives. At MECLABS, our research analysts do this visually using a small flowchart to represent the flow of traffic to the control and treatment pages.
Take your latest research question and write it down. Below it, write out the following until you have listed all the variations to be tested.
At MECLABS, our analysts use the Test Protocol document to determine how many site visits are required to achieve valid results given a set of treatments and typical conversion rates on the existing page. It is very important that traffic is randomly split between the treatments and control. In a high traffic site, the percentage sent to the control can be higher than what is sent to the treatments, as long as you will easily meet the required minimum sample size.
Multi-factorial tests
The split test design works for tests of only one step, but sometimes we need to test more than one step in a process. We have two independent variables that we will manipulate separately. For example, if your research question is, “Which checkout process generates the most revenue?” you might want to test several variations of cart layout and payment page layout at the same time.
To test multi-step processes, researchers use a research design called a factorial test. Each variation in each independent variable is tested together so that all combinations are tested.
Traffic volume is crucial for factorial tests
One common reason some marketers don’t run multi-factorial tests is a low-traffic page. For example, with only 3,000 hits a month, a 7% historical conversion rate, and six treatment pairs (2 payment designs x 3 cart designs), it could take as much as three years to validate the factorial design shown above!
When faced with an unreasonable completion time, you have a few choices to make. You can test fewer treatments, resulting in quicker accumulation of hits on each treatment, or you can test one step of the checkout process at a time.
Diana Sindicich is a data analyst in the Sciences Group of MECLABS.
