After reading about the huge conversion boosts many bloggers got after A/B testing their headlines, I assumed A/B testing the headlines of 73 articles I have published on my language learning blog French Together would be a no brainer.
I could already picture myself writing a triumphant article showing how A/B testing my headlines helped me double my traffic and get an extra 3 million visitors a year.
Unfortunately, the magic never happened and the article you are about to read will instead share the 5 lessons I learned along the way.
Lesson #1 A/B testing won’t work for everyone
A/B testing sounds easy. You write 5 article headlines, wait a few days, choose the winner and enjoy a 132% visitor increase.
One aspect many forget to mention is that you need lots of visitors to show your headline variations to.
And I really mean a lot.
French Together received 250,757 visitors last month and even with these numbers, 19 headline A/B tests (out of 73) still don’t have a clear winner after almost 4 months.
Some of them because the new headline simply wasn’t different enough, others because the article didn’t get enough traffic.
There is no clear answer to how much traffic is needed to run an A/B test because it depends on lots of factors such as the conversion rate of your existing headlines but I wanted to give you a rough number anyway so I used Optimizely’s A/B test sample size calculator.
With a baseline conversion rate (the average conversion rate of my existing headlines) of 50% and a statistical significance of 99%, Optimizely’s A/B test sample size calculator estimates that I would need 200 visitors per variation to run an A/B test with a statistical significance of 99%.
This means testing the headline of an article with 100 views per month would take 4 months if you only A/B test one variation and 8 months if you have 2 variations.
So before you A/B test your headlines, make sure you have enough visitors to get statistically significant results.
If you don’t have enough visitors, all is not lost!
You can still use a tool like Priceonomics Content Tracker which uses Facebook ads instead. This isn’t free but it will allow you to get results faster since your leverage Facebook’s audience instead of yours.
Lesson #2 Don’t be as naive as I was
My headline A/B test was a huge disappointment because I had incredibly naive expectations.
I honestly expected that most of my A/B tests would not only give me quick results but also lead to tremendous conversion increases.
After more than 3 months, and 73 headline tests, the results are a lot less impressive than I had hoped:
- 19 tests are still running with no conclusive results
- 31 tests where the initial article headline performed better than the variations I tested.
- 23 tests where at least one of the variations beat the original headline.
This means only 23 A/B tests out of 73 actually resulted in a higher-converting headline
This is not a bad result and could still bring thousands of new visitors to French Together but this is far from the 486% improvement many bloggers talk about.
Lesson #3 It’s much harder to go from good to great than from bad to good
The results I just shared with you are disappointing but they shouldn’t have surprised me.
I have always considered great headlines essential so the article headlines on French Together were already pretty good and turning good headlines into great article headlines is much harder than turning mediocre headlines into good headlines.
If your website has purely descriptive and rather boring blog post titles like “Summer vacation” or “broccoli recipe”, you are likely to see a much bigger conversion boost simply because there is a lot of room for improvement and a lot of mistakes to correct.
Lesson #4 There is a fine line between a great headline and clickbait
One thing I realized while trying to come up with better headlines was that there is a fine line between a great headline and a clickbait headline.
It’s tempting to promise a lot in your blog post title because the greater the promise, the more clicks you are likely to get.
But that would be forgetting that the goal of your headline isn’t simply to get more readers, it’s to attract readers who will value your content and trust you.
Getting rid of potentially high-converting headlines just because they aren’t entirely truthful is painful but it’s necessary if you want people to trust you.
“15 strange French sayings they don’t teach you at school” sounds like a cool blog post title.
It’s intriguing, it makes you want to click.
But do I know for sure that the 15 sayings aren’t taught in the reader’s school?
How will the reader feel when he/she reads the article and realizes they already know all these sayings?
Lesson #5 Balancing SEO and copywriting is a difficult exercise
The #1 result for “French expressions” doesn’t contain the keyword “French expressions” but Google is smart enough to understand the topic and rank it #1 anyway.
One particularly tricky aspect of SEO is the fact that there sometimes seems to be a huge difference between what’s good for your readers and what’s good for search engines.
We are told that it’s better to place a keyword in the headline, at the beginning even.
But it feels pretty awkward sometimes and the last thing we want is for the content of our website to look like it was written by R2D2.
So what do you do?
As often, the key is balance.
Start by writing down the main keyword you want your article to rank for.
Then see if you there is an elegant way to add this keyword to your headline.
If you can, great!
If you can’t, it’s okay too. Your article can rank for a keyword even if the keyword is not mentioned in the blog post title. In fact, I frequently see articles that rank for keywords they don’t mention at all.
You are more likely to rank for a keyword if you mention it in your headline but this shouldn’t come before your readers.
Because, remember, Google only cares about one thing ultimately: making sure people who go to your website enjoy the content. Your website won’t stay number one for long if Google sees that readers immediately leave it and look for other results.
Note: the keyword you put in your headline doesn’t have to be absolutely identical to your main keywords. Google is smart enough to know that a headline mentioning “best podcasts” should also rank for “best podcast.”
Conclusion
This headline experiment didn’t end up the way I hoped it would but it taught me great lessons and I hoped you learned a lot from it as well.
I still believe in the value of A/B testing. I have run hundreds of A/B tests over the years and testing my optin forms and sales pages helped me double my conversion rate on several occasions.
But A/B testing is not always the right choice.
So should you A/B test? Or should you keep your current article headlines?
My advice is simple: A/B test if your current blog post titles are terrible and you have enough traffic (or money). If not, I recommend downloading the headline checklist and writing the best headlines you can.