Article

Roland Pokornyik
Roland Pokornyik 11 March 2016

5 Mistakes That can Ruin Your Email A/B Tests

Testing your templates, your copy, your timetable can be a great help, but most of the time it won’t give you the Holy Grail. You have to test major and minor changes also in order to narrow your email down to the perfect version.

Expecting instant and spectacular results

But if you change just something small in most cases the difference will not be so spectacular that you can tell it truly is the alteration that made a difference or some other factor.

It might take a while till you get statistically significant results, especially if you have a small email list. You may need to repeat your tests various times to be able to make a well established decision. You don’t have to be a pro in statistics, but an A/B split test significance calculator might be quite handy.

Not testing at the same time

If you have multiple versions of your email, you should test them at the exact same time. Except when you are testing for the ideal time to send your mail. In this case you should use the exact same template, copy, subject line and so on. The one and only difference should be in timing.

If you don’t test different versions simultaneously you risk getting inconsistent results. Recipients won’t react to your emails the same way at different times. It definitely matters if you send your emails in the morning or the afternoon, on the weekend or in the middle of the working week. Be really specific about the goals of your test, and thegeneral rule of thumb is to modify only a single element at the same time.

Running too many tests

It’s great news that you want to improve your email campaigns, and definitely A/B tests are a great way to do that, but you shouldn’t overuse testing. Too many tests mean overwhelming amount of data, which can be confusing and impossible to analyze in its entirety.

You can and should absolutely test for important changes, but keep the number of simultaneously running tests low so you can interpret the results properly.

Defining the wrong KPIs

Before you even think about starting a test, define it clearly what is the goal of your test, and how are you going to measure the results. A small difference between templates can increase your click through rate, but in the end you may end up with lower conversion (purchase, order, whatever) rate. You may be looking at decreasing open rates while missing the fact that simultaneously more people click on your links in the mail.

In most cases the final conversion won’t happen inside the email, so you have to be able to track your website properly to be able to make the right decisions.

Not testing your list right

If you make extreme changes to your email template, test with a smaller number of subscribers first – this way the unsubscribe rate will be less drastic if it goes wrong. This is also true for offers where you want to get the most conversions in a limited time: test only a (random) portion of your list (or segment) so you can refine your template and copy.

Obviously there are tons of other tips we should list here. But maybe a definitive guide to email A/B testing would make the most sense. What do you think? Would you like to read about the depths of A/B testing?

Please login or register to add a comment.

Contribute Now!

Loving our articles? Do you have an insightful post that you want to shout about? Well, you've come to the right place! We are always looking for fresh Doughnuts to be a part of our community.

Popular Articles

See all
Digital Marketing Vs. Traditional Marketing: Which One Is Better?

Digital Marketing Vs. Traditional Marketing: Which One Is Better?

What's the difference between digital marketing and traditional marketing, and why does it matter? The answers may surprise you.

Julie Cave
Julie Cave 14 July 2016
Read more
4 Important Digital Marketing Channels You Should Know About

4 Important Digital Marketing Channels You Should Know About

It goes without saying that a company can't do without digital marketing in today's world.

Digital Doughnut Contributor
Digital Doughnut Contributor 5 November 2014
Read more
A Look At Top Mobile App Development Trends for 2017

A Look At Top Mobile App Development Trends for 2017

The demand for apps is increasing at a rapid pace and there are some mobile app development trends to watch out for in 2017.

Alesia Nik
Alesia Nik 18 January 2017
Read more
Android vs. iOS: Smartphones, Which And Why

Android vs. iOS: Smartphones, Which And Why

You’re in the market looking for a new smartphone. Here's the question, which smartphone OS should you invest your money in?

Jenny Granados
Jenny Granados 19 January 2017
Read more
Year’s Best Tech Gadgets at CES 2017 Revealed

Year’s Best Tech Gadgets at CES 2017 Revealed

The biggest tech event of the year, CES 2017 showcased a wide assortment of new gadgets, Cars and TVs that will leave you spellbound.

Alesia Nik
Alesia Nik 16 January 2017
Read more