Article

Roland Pokornyik
Roland Pokornyik 11 March 2016

5 Mistakes That can Ruin Your Email A/B Tests

Testing your templates, your copy, your timetable can be a great help, but most of the time it won’t give you the Holy Grail. You have to test major and minor changes also in order to narrow your email down to the perfect version.

Expecting instant and spectacular results

But if you change just something small in most cases the difference will not be so spectacular that you can tell it truly is the alteration that made a difference or some other factor.

It might take a while till you get statistically significant results, especially if you have a small email list. You may need to repeat your tests various times to be able to make a well established decision. You don’t have to be a pro in statistics, but an A/B split test significance calculator might be quite handy.

Not testing at the same time

If you have multiple versions of your email, you should test them at the exact same time. Except when you are testing for the ideal time to send your mail. In this case you should use the exact same template, copy, subject line and so on. The one and only difference should be in timing.

If you don’t test different versions simultaneously you risk getting inconsistent results. Recipients won’t react to your emails the same way at different times. It definitely matters if you send your emails in the morning or the afternoon, on the weekend or in the middle of the working week. Be really specific about the goals of your test, and thegeneral rule of thumb is to modify only a single element at the same time.

Running too many tests

It’s great news that you want to improve your email campaigns, and definitely A/B tests are a great way to do that, but you shouldn’t overuse testing. Too many tests mean overwhelming amount of data, which can be confusing and impossible to analyze in its entirety.

You can and should absolutely test for important changes, but keep the number of simultaneously running tests low so you can interpret the results properly.

Defining the wrong KPIs

Before you even think about starting a test, define it clearly what is the goal of your test, and how are you going to measure the results. A small difference between templates can increase your click through rate, but in the end you may end up with lower conversion (purchase, order, whatever) rate. You may be looking at decreasing open rates while missing the fact that simultaneously more people click on your links in the mail.

In most cases the final conversion won’t happen inside the email, so you have to be able to track your website properly to be able to make the right decisions.

Not testing your list right

If you make extreme changes to your email template, test with a smaller number of subscribers first – this way the unsubscribe rate will be less drastic if it goes wrong. This is also true for offers where you want to get the most conversions in a limited time: test only a (random) portion of your list (or segment) so you can refine your template and copy.

Obviously there are tons of other tips we should list here. But maybe a definitive guide to email A/B testing would make the most sense. What do you think? Would you like to read about the depths of A/B testing?

Please login or register to add a comment.

Contribute Now!

Loving our articles? Do you have an insightful post that you want to shout about? Well, you've come to the right place! We are always looking for fresh Doughnuts to be a part of our community.

Popular Articles

See all
Digital Marketing Vs. Traditional Marketing: Which One Is Better?

Digital Marketing Vs. Traditional Marketing: Which One Is Better?

What's the difference between digital marketing and traditional marketing, and why does it matter? The answers may surprise you.

Julie Cave
Julie Cave 14 July 2016
Read more
4 Important Digital Marketing Channels You Should Know About

4 Important Digital Marketing Channels You Should Know About

It goes without saying that a company can't do without digital marketing in today's world.

Digital Doughnut Contributor
Digital Doughnut Contributor 5 November 2014
Read more
Are virtual reality and augmented reality the next step for CMS?

Are virtual reality and augmented reality the next step for CMS?

In a fitting conclusion to a year many hailed as the “year of virtual reality” (VR), in December 2016, WordPress made it possible for users to create and publish VR content on any WordPress site. The change meant that users could publish 360-degree photos on their sites, and regular photos and panoramas could be viewed in VR.

Leonie Mercedes
Leonie Mercedes 16 October 2017
Read more
These 5 Changes Are Coming For Your Site in 2018. Are You Ready?

These 5 Changes Are Coming For Your Site in 2018. Are You Ready?

Let's face it, the Internet is ever-changing. Some changes you can’t predict: at the start of 2011, no one realized that responsive would practically be a requirement by the end of the year. But some changes are known and scheduled. Let’s look at some of the biggest changes that may affect the way your company communicates through digital channels.

Mike Norman
Mike Norman 13 October 2017
Read more
Collection Of The Best Email Testing Tools Online

Collection Of The Best Email Testing Tools Online

Don’t be afraid of email testing. There are many free or freemium tools online that can help you with testing your SPAM score, deliverability and even the rendering of your email. We feature 30 email testing tools in this article. Check out the complete list!

Roland Pokornyik
Roland Pokornyik 31 October 2016
Read more