You love the artwork of Van Gogh and the excitement of NASCAR. It’s baffling to you that others don’t share your opinion equally and with the same fervor. While sticking to your own preferences in your private life is just fine and is even grounds for playful arguing at parties, in the business world, you need to make personal connections with people as often as you can. Or, get as close as possible.

Your in-house creative staff is the bomb! But they have their preferences and biases too. What we don’t want is to lose an opportunity to attract more customers because we stubbornly stick to one position on our look, feel, or content. What do the customers want? And how do we find this out through testing?

What Is A/B Testing? 

A/B testing is a method of comparing two versions of a web page (apps and digital ads can also be A/B tested) against each other to determine which one performs better. A/B testing is essentially an experiment where two or more variants are shown to users at random, and statistical analysis is used to determine which variation performs better for a given goal.

Testing takes the guesswork out of website optimization and enables data-informed decisions that shift business conversations from “we think” to “we know.”

By measuring the impact that changes have on your website metrics, you can ensure that every change produces positive results.

How Does A/B Testing Work?

You have two different designs for a website landing page, and you want to know which one will perform better.

After you create your designs, you give one landing page to one test group and you send the other version to the second test group. Then you see how each landing page performs in website metrics such as traffic, clicks, or conversions.

If one performs better than the other, great! You can start digging into why that is, and it might inform the way you create landing pages in the future.

For Example

A company with a customer database of 2,000 people decides to create an email campaign with a discount code in order to generate sales through its website.

The company creates two versions of the email with a different call-to-action.

The company sends the email to 1,000 people with the call-to-action stating, “Offer ends this Saturday! Use code A.”

And to another 1,000 people, it sends the email with the call-to-action stating, “Offer ends soon! Use code B.”

All other elements of the email’s copy and layout are identical. The company then monitors which campaign has the higher success rate by analyzing the use of the promotional codes.

The email using code A has a 5% response rate (50 of the 1,000 people emailed used the code to buy a product), and the email using code B has a 3% response rate (30 of the recipients used the code to buy a product).

The company therefore determines that in this instance, the first call-to-action is more effective and will use it in future sales content.

What Are the Benefits of A/B Testing?

The overall objective of A/B testing is to make it easier for a user visiting your website to follow the journey that you want them to take. Possible benefits include:

·      Better content engagement on site

·      Lower bounce rate

·      Higher email or phone inquiries

·      Increased conversions

While A/B testing focuses predominantly on the user’s experience, it’s important to always keep SEO top of mind.

Some Inherent Problems with A/B Testing

You’re testing your own assumptions. So, allowing bias to distort your A/B tests, or mis-framing, is both common and frequent.

It takes a long time to see results (minimum of 3-4 weeks). That opportunity cost could be better spent somewhere else.

Results are going to be dependent on a meaningful sample size. So, it might take a while to attain true statistical significance. You want to see numbers you can trust.

Changing too many things at the same time makes it difficult to know which change influenced the test. Test components one after the other so results won’t get conflicted. You need to prioritize and solve the most critical one first.

When A/B Testing Doesn’t Make Sense

When the web pages being tested are not receiving enough website traffic.

The average amount of website traffic for a page to be testable in a reasonable amount of time is usually around 30,000 monthly visits. That’s not to say that you can’t test a page with lower traffic, you just have to be prepared for the tests to run for a longer period of time. Sometimes it can take years for pages with less than 1,000 visits a month.

When the web pages are not experiencing enough conversions.

The higher the conversion rate, the faster the test will reach significance. Additionally, the larger the difference between the conversion rates of the variations, the faster the test will reach statistical significance. A conversion rate above 10% is often a good place to start.

When the web page doesn’t have the potential to have an impact.

This is arguably the most important part of deciding whether to test. If the web page has the potential to have a significant impact on the bottom line of your business, then the opportunity to test is more appealing. However, if the web page isn’t a strong contributor to revenue generation, it’s more likely that the cost of testing will outweigh the benefits.

A/B testing can be a valuable resource or it can produce inconclusive results. The value of it depends greatly on the amount of website traffic you can generate so that any statistics you view will be meaningful and truly inform your decision to choose option A over option B or vice versa.

A global team of digerati with offices in Washington, D.C. and Southern California, we provide digital strategy, digital marketing, web design, and creative for brands you know and nonprofits you love.

Stay up to date with our insights by following us on Twitter, Facebook, and LinkedIn.