Phrasee | Testing for success

TESTIN G F OR SUCCESS

Testing, testing... testing? Testing is the secret to better optimization and more ROI. Organization is testing to understand what language is resonating: according to our survey of 300 senior US marketers) Put simply, it’s how marketers can find out what’s ( resonating, and then double down on it. We don’t like to scaremonger, but if you’re not doing any meaningful testing as a marketer, you’re essentially sabotaging your chances of growing engagement. 16% 24% The problem is this: less than one in four marketers is doing any type of testing to understand what is resonating with customers, according to our survey of 300 senior US marketers. Of those that are, the majority is only doing Yes rudimentary A/B testing. If this rings true for you, then this 12% No, it’s too time consuming paper is your springboard for putting things right with a but we would like to meaningful testing strategy... No, I don’t think it is important I don’t know 48% www.phrasee.co 2

Jen Capstraw, Co-Founder of Women of Email, on the importance of testing with meaning: More focus on testing and optimization can produce two possible outcomes. Possibility One: Improved customer experience which boosts the bottom line. (Hooray!) Possibility Two: A lot of spinning of wheels that have little impact on top-level business goals. Marketers have a pesky habit of hyper-optimizing campaigns with limited impact, or creative minutia that doesn’t matter all that much in the grand scheme of things. Testing can produce big gains if you have informed talent and technology that enables optimization at scale. But otherwise, it can be a real time-waster. Instead of going through the motions of a test because it’s “best practice” to “test everything,” zero in on where you can make the most meaningful business impact. Where would a subtle improvement actually move the needle? (Tip: It’s definitely not testing the color of a CTA button in a one-off campaign.) Once you figure out what might make a difference, you want to design a testing strategy that minimizes opportunity cost. That means, the least-effective versions of your tests go to the smallest groups of folks possible while still ensuring reliable, meaningful results. And finally, tests should provide learnings that you can apply going forward. Speaking of which - this Orange Paper is your perfect jumping off point for a meaningful testing strategy!” www.phrasee.co 3

7 steps to conducting successful split testing So how do you move beyond A/B and get good results based on sound When improving email, or indeed the performance of any marketing channel, most people will conduct a basic two-version split test, i.e A/B data? It only takes 7 easy steps. testing. It's quick, it's easy – so why go beyond that? Because when you test more combinations and more variables, you get more data, and data is how you make your email program better. If you test 10 different subject lines variants, you are testing a much wider space of possible language. 1 Set your goals. We’ll focus on subject lines for the purpose of this paper, but the same principles can apply to all manner of marketing copy: preheaders, CTAs, body copy… the works! 2 Set your test group. The best brands in the world are doing loads of split testing, and it's making a huge difference: 3 Allocate the splits. 57% 30% 4 Generate your copy. click rate uplift open rate uplift 5 Test, then wait. 275% $millions ROI increase in incremental revenue 6 Get (and share) the results. 7 Identify (and learn from) patterns. 102% increase in campaign revenue www.phrasee.co 4

Step #1: Set your goals. Step #2: Set your test. The biggest mistake marketers make is testing when they shouldn't - Set aside a random segment of your audience. This will be your "test" without meaningful or clearly defined goals in place. group who will see the variants you're testing and allow you to gauge audience response to your messages. To avoid this aimless limbo we recommend starting testing with your subject lines. Numerous sources of research - which you can find in The most important thing about your test group is making sure it's large our blog on the subject - show that more opens means more clicks enough so that the test results you get are statistically significant (see and conversions. Plus, you can get data from subject line testing that our definition below), but small enough so that there's still plenty of you can then apply to the rest of your email content later. room for the winner to go out to the control group at the end of the test. So if you're testing email subject lines, your goal - the KPI that you're Using 30% of your total database for split tests is a reasonable trying to move the needle on - is your engagement. Engagement is assumption for average-sized datasets, but for particularly large measured by open, click and conversion rates, so it’s up to you which of datasets, you might consider decreasing the percentage to optimize these metrics you choose. potential revenue once you’ve identified your highest-performing subject line. It’s worth remembering that Apple’s Mail Privacy Protection measures have introduced some statistical noise to open rates, so some marketers For a shortcut to setting your test group based on your audience have shifted their focus towards clicks and conversions. size, head to our split test calculator! Check out our Orange Paper on what you can do to mitigate the impact of Mail Privacy Protection. What is statistical significance? Statistical significance (or "stat sig") is the likelihood that a relationship between two or more variables is caused by something other than chance. Subject line split testing with a large enough group of recipients will get you results that are reliable enough to draw conclusions from. Test with small groups at your peril! All you'll get is a few random, largely meaningless data points. www.phrasee.co 5

Step #4: Generate your copy. Step #3: Allocate the splits. If you've decided to test 10 different variants, then you'll need to come up with 10 different subject lines, which are often the shortest bit of copy in the email... and often the hardest to write! So what makes good subject lines? Determine the number of variants you'll test and how many people in the test group will get each one. Remember that each group The main ingredient is diversity. Using the same language repetitively decreases copywriting performance over time, meaning less needs to be large enough so that we have statistical significance in the results! As a general rule, your split test sizes should be in the engagement and less revenue for your brand. So mix it up as best you tens of thousands. can – change the tone of voice, the punctuation, try capitalizing some words, and sprinkle in some emojis! The more varied they are, the more Now, at this point we could get into bionomial vs Gaussian varied data you’ll get to learn from during your test. distribution, bell curves, and standard deviations from the mean, but instead, we just put together this split test calculator tool! It makes it easy to choose the right size for your split test segments based off of things like science and math! Gaussian distribution bell curve www.phrasee.co 6

Step #6: Get (and share) the results. Until the testing window has passed, go see which variant gave you the best result based on the metric you want to optimize for. That’s the winner! Now you can send that one out to the rest of your audience. Let’s say that one resulted in a 6% click rate while all the others were in the 2-3% range. Now go tell your boss that you used split testing to Step #5: Test, then wait. achieve a 3-4% increase in click rates, and then maybe casually ask when the next bonuses are going out! (Wink!) Time to send out your emails! Each one should have identical content This can't be right... inside with a different subject line. Once you've hit send, now you've got some time to kill (at least four waking hours is what we recommend The results of any email subject line split tests should all be within the during a high-traffic time - you won't get actionable results if people same range. If you see results that look like: aren't looking at their emails!). Bake a pie, catch up on your inbox, or negotiate a peace treaty if you're feeling ambitious! But whatever you 12%, 13%, 12%, 42%, 12% do, don't declare a split test winner until the testing window has passed because you may get skewed results. It doesn't mean that you've just found the most effective email subject line in the history of the world, it means that something has gone wrong. Technical errors, bugs and bots can cause havoc with your results. No need to lose sleep - but it’s advisable to try and diagnose the cause of outliers so that you can decide whether or not to include them in your reporting (and exclude from future tests). www.phrasee.co 7

Step #7: Identify (and learn from) patterns. Looking at the results of a single test will only go so far. The real goal here is to learn over time. We all have theories about what does and doesn't work in an email subject line, but the data will confirm or deny! Take the results of your split tests together and start identifying patterns. For example: do you need to change your emoji usage? Shorten your copy? Alter your sentiments? Before long, the fine points of your brand language will be polished for your audience. Once you see what's working (and not working), you can use that info to make smart decisions about what's inside your emails next. "This sounds like a lot of work" (Or your text messages! Push notifications! The sky's the limit when it comes to split testing.) You're right – it is! But it doesn't have to be. Phrasee makes split testing as easy as clicking the Magic Button. We generate human- sounding subject lines custom-tailored to your brand – and your brand only. Then, we test, learn, and share the data with you so that your whole team can say it better. We've got experienced language technicians combined with the power of AI, and we use it to produce incredible results for the world's biggest brands. Want to see how easy it can be? Watch this 1-minute video – you'll be blown away! www.phrasee.co 8

Empower your marketing with Brand Language Optimization. Book a demo US: (415) 941-2420 UK: +44 (0)20 8870 6968 [email protected] www.twitter.com/phrasee www.linkedin.com/company/phrasee 7 www.phrasee.co