A/B testing is the process of running an experiment between two or more pages simultaneously to see which performs or converts the best. Despite the name (A/B testing), the experiment can be conducted with as many pages as you want. As soon as you have chosen which elements you want to test on your landing page (e.g. call to action, photography, headline, submitting a video etc.) – you can create new versions (or variants) of your page to enter into the experiment.
When you have finished testing, you determine which one is the winner (the page that has the best conversion performance). This is referred to as the champion page. When you begin a test, you create new versions (variants) in order to challenge the champion page that already exists. These are referred to as challengers. ‘Page variants’ refers to any new version of your landing page that is included in the test. The champion and any challenger pages are all variants.
Assigning traffic weight in an A/B test
Traffic is randomly assigned to every page variant based upon a predetermined weighting – for instance, if you are running a test with 2 page variants, you may well separate the traffic 50/50 or 60/40. Visitors are generally cookied so that they will always see the same version of the page (in order to keep the integrity of the test). The pivotal factor which decides how much weight you would ascribe to your page variants during a test is timing – whether you are beginning the test with multiple variants at the same time or testing new ideas against an established page.
Starting with multiple page variants
If you’re beginning a campaign and have many different ideas about which direction to take, you can create a new landing page variant for every idea. In this scenario you would most likely assign equal weight (traffic) to every page. The reason is that you want to treat them equally and select a winner (champion) as soon as possible. It’s important that you drive a certain amount of traffic through test pages before the results are statistically valid or important, and as you do not have any conversion data on any of the pages, its logical to start your experiment from a position of equality.
Testing against a pre-existing page
If you have an established page that you want to try some new ideas out on, you would start by giving your new page variants a smaller percentage of traffic than the existing champion to analyse the risk included when introducing new ideas (which may not perform well).