Sometimes It’s Not You, It’s Them: When CRO Testing Gives Results You Didn’t Expect

In my years of conversion rate optimization testing I’ve seen some remarkable test results, remarkably good, and, unfortunately some remarkably bad.

Who Doesn’t Want Free Shipping?

I ran a test for a major cosmetics brand. If the visitor had 15 minutes of inactivity they were served a timeout page with the headline “RETURNING CUSTOMERS” and then below that in much smaller font, a message saying the visitor has timed out and they need to sign back in. There was nothing in the experience that gave a reason WHY a customer should login again. 

So, the alternative to be tested touted some of the advantages of continuing to shop. This cosmetics company was offering a free sample available at checkout and free shipping on all orders. Free shipping? Everybody loves free shipping! Certainly reminding, or informing for the first time, that free shipping was available was going to move numbers in the positive direction, right? Free shipping and a free sample should remind the users of the great value they’re getting, with that reminder we’d hope they’d sign back in and continue the purchase funnel. 

Amazingly, the new timeout message saw a 8.2% decline in conversion rate, and revenue per session decreased 10.2%.

Why? When a breakout of the data was examined, it turned out that the negative effects were almost entirely with customers from China and South Korea. Separating the traffic from the United States and the test variation had no effect. The best guess of what may have happened is that the new language was confusing to Asian customers who may have become accustomed to the control message, the U.S. audience probably already knew about free samples and free shipping so it had no effect upon them.

The Site That Couldn’t Be Moved 

Another client of mine had a website with an unusual value proposition: if you answer TV “trivia” questions about shows that recently aired you could be put into a sweepstakes where you have a very slight chance of winning something. To get to the trivia questions, the user had to fill out a form with a lot of personal information: do you own or rent your home? What race are you? Do you have children? Pretty nosey stuff.

And those trivia questions? Half of them were about the advertisements that aired during that time. An example of a trivia question might be “In the (laundry product) commercial do you remember if the (product)’s value was it made whiter whites or keeps colors faster and longer?” This data would then be sold back to the advertiser as customer research.

We tried all kinds of tests to increase the number of users past the intrusive form, and into the trivia game. Nothing worked. We tried social trust tests, we tried pointing out some of the cooler prizes that could be won in the sweepstakes and we even redesigned the old form because it was still coded in <table> tags. Not a single test provided lift.

I was very dispirited and judging my skills as an optimizer. Then one day, the client mentioned that 95% of inbound traffic was from affiliate sites and those sites were all featuring offers for free things. So, unfortunately, the audience flowing into the tests didn’t care what the UX was, didn’t care if they had to fill out a 20 field form, all they wanted was a chance to win a gift card or a kitchen blender.