How Coderwall grew 57% with one A/B test

Dillon Forrest, Cofounder of RankScience Dillon Forrest updated this post on January 7th, 2016

RankScience automates SEO services for companies, including SEO Split Testing. Take the guesswork out of SEO and get data-driven.

A/B test background: Coderwall

Coderwall is a community for developers to share development & design tips, tools, and projects. They wanted to take a data-driven approach to user acquisition via organic search.

From an SEO perspective, Coderwall was already pretty advanced, with a rich backlink profile and 130,000 pages of user-generated landing pages generating organic search traffic.

Most of Coderwall’s organic search traffic comes from their long-tail pro-tip pages. Users create pages to share programming tips with one another, such as JavaScript performance tips or Ruby on Rails tutorials. In fact, of their top 500 pages most visited by users from Google, 499 pages are pro-tips (the single page which was an exception was their homepage).

The key to Coderwall’s growth was earning more organic traffic on these pro-tip pages.

How can we get more clicks via organic search?

On average, most of Coderwall’s pages in Google were ranking between bottom of the first search engine results page (SERP) or top of the second SERP.

Coderwall SERP before a/b test

Coderwall landing at #8 on page 1

Landing anywhere on page 1 of a Google search is obviously great for maximizing your search impressions, but being on the bottom of the first page generally earns only a small percentage of clicks. Low clicks and high impressions translates to a poor clickthrough rate when you’re at the bottom of the SERP. The top 3 search results get 63% of all clicks in search engines. The bottom 3 results (#7-10) get only 9% in total, even though they’re still on the first page. (Note: studies show that only 10-15% of clicks go to Adwords links, even though Google is showing more ads on the SERPs these days.)

Clickthrough rates for each link on a SERP

There are two ways in which you can increase organic search traffic: 1) improve CTR on your pages without improving their position on the SERP, and 2) increase your positions on the SERP. Your position on the SERP is determined by Google’s search algorithms. A lot of companies are afraid of rolling out changes to pages that are already ranking and getting traffic from Google, fearing that they might displease Google’s search algorithms, hurt their position on the SERP, and break something that’s working.

The titles of search results are the clickable blue links, and titles that are more persuasive generate more clicks. Our hypothesis was that a simple <title> tag change could increase CTR and fatten the top of the marketing funnel, extracting more clicks from Coderwall’s existing positions in Google.

Building the A/B test for SEO

When most people think of A/B testing, they think of conversion rate optimization (CRO) and tools like Optimizely. Since Optimizely and other A/B testing tools for CRO are meant to fatten the middle of your marketing funnel, and since Coderwall wanted to fatten the top of their funnel instead, Coderwall needed different tools.

We built RankScience as a CDN proxy to implement A/B testing experiments for SEO for brands like Coderwall. After we helped Coderwall run a few A/B tests, the biggest win for Coderwall was a <title> tag change.

The original title format was just “{Post of title} | Coderwall”. The proposed new title format was “{Post of title} (Example) | Coderwall”.

A/B test variations side by side

Side-by-side comparison of the original and new title formats

Roughly 20,000 HTML pages were split into two groups, the control group and the variant group. The control group’s title had the original format, and the variant group’s title had the new format.

We measured the two group’s performances before implementing the new format for the variant group. After the new format was deployed to the variant group pages, we waited three weeks to gather new data.

A/B test results

While most of Coderwall’s A/B tests failed to produce a significant uplift, the title tag experiment was a huge win. The number of users clicking on Coderwall from Google increased 14.8% (yes, this was statistically significant).

Chart of uplift

Uplift from the title experiment

With data at their side, Coderwall felt confident deploying this change to all of their pro-tip pages. As expected, the new title format continued to grow their search traffic over time, even months after concluding the experiment.

Coderwall search traffic up and to the right after several months

Coderwall traffic over the span of multiple months

To date, Coderwall’s increase in traffic compounded to an aggregate 59% from this experiment alone.

What did we learn from this experiment?

We learned that users who wanted Coderwall’s pro-tip content wanted tactile information that was easy to digest and immediately applicable to their own work. These extra details exposed by “(Example)” persuaded users to click on Coderwall links instead of others on the SERP. Over time, Google noticed the CTR for Coderwall pages going up, presumed users were finding quality results for what they wanted, and rewarded Coderwall with improved position and impressions, growing 8% month over month.

How is this useful for you?

Does this mean you should run out and try appending “(Example)” to the title of all your pages? Probably not.

This test came about in the context of several other concurrent A/B tests. Analyzing all the test results exposed key learnings about Coderwall’s users.

To apply this to your business, you should be running your own SEO experiments to test your hypotheses and learn more about your users’ intentions. This works for Coderwall because it aligns with their user intent. The tests that will work for you will align with your own users’ intent.

→ Are you interested in seeing more a/b testing case studies?

RankScience is taking signups if you’d like to start split testing your SEO today! We split test SEO for everyone from startups to Fortune 500 companies.

Learn More About RankScience

We are engineers-turned-SEOs, and we built RankScience to enable companies to run SEO experiments. If you want to learn more about our data-driven SEO platform, we’d love to chat! [email protected].

About the author

Hi everybody! I'm Dillon, Cofounder at RankScience. I focus on making data-driven SEO recommendations with A/B tests.

You can email me with any questions or tweet me to let me know what you think of this post!