You’ve probably seen a lot of blog posts floating around the internet about A/B testing successes. This blog is no exception.
But hardly anyone talks about their failed experiments. I don’t blame them–it’s hard to expose to the world that you weren’t right about something.
Which leads us to believe…is anyone running failed, insignificant tests? I mean, if no one’s talking about it, it must not be happening, right?
Let me tell you a secret: Everyone is failing. Everyone is running or has run an experiment that got them nowhere.
However, at Kissmetrics, failure is part of our A/B testing process. If none of our tests fail, we know we’re not running enough tests or our ideas are too safe.
In fact, the bigger the failures, the closer we are to an even bigger win.
We’re never 100% correct about our hypotheses. No matter how many years of experience you have, no matter how much you think you understand your customers…there’s always room for learning.
Otherwise, why would we test in the first place?
Now let’s take a look at a couple of our own failures so you can see what I mean.
Failure #1: Too much white space on our product page
Test Hypothesis:There’s too much clutter at the top of the page. By removing the background image and reducing white space, we’ll make the page copy more visible, enticing people to scroll down and interact more with the page.
You already know that this test failed, but just from looking at the hypothesis–do you know why?
I’ll give you a hint: it has a lot to do with data.
We technically had data that indicated a dip in conversion on this page.
However, we didn’t have evidence that people weren’t scrolling down, or that the space at the top was stopping them from converting. When a hypothesis has little or no evidence, we have a slim chance of winning the test.
Improvement over original: 4.41%
Certainty: 55.27% for the variant
What we learned:
No significant data here.
Having a hero image or not in this page won’t influence our conversions. In previous homepage tests, our hero image mattered, but perhaps not on this exact page.
In previous tests on this particular page, we experimented with the copy and overall messaging. Therefore, our next test should be around the copy to see if we’ll get a lift.
Failure #2: Copy and images on the product page
Test Hypothesis:A more benefits oriented product landing page will lead to better quality leads.