At Searchspring, we try to make all decisions using data. Data-driven decisions are the key to success and delivering what our customers want. If we built all the cool things we wanted to build, we wouldn’t need a product team. We would let Engineering just build cool things, which would be a terrible idea.
In order to make data-driven decisions, you must be able to test your decisions in a way that is fair. I see too many retailers out there showing before and after data and saying: “see, it worked”. But did it really? Are your before and after data sets under the exact same conditions? Are your sales perfectly consistent? Do you never or always run the same promotions? Was the weather the same?
You see, there are way too many variables to ever use a before and after chart to prove your point and show success. The only valid measure is to do an A/B test to determine which outcome is statistically better. I worked at a company where we made a change to the product, and when we A/B tested it there was absolutely no difference. According to the statistics calculator, it would take no less than 10 years to declare a winner. Was that change worth it? No way, it used up precious engineering resources and provided exactly no marginal benefit to our clients.
So why don’t people use A/B testing more often? It’s difficult, like really really difficult, not to mention expensive. Additionally, there are times that A/B testing is pointless. Let’s say you are going to change platforms, you’ve already made the decision. Would A/B testing work in this scenario? Could you afford to run both platforms side by side? Could you handle the order management from two different systems? The answer to both of those is probably a good hard no. So you made the decision and you went with it, that’s fine. You don’t get to second guess yourself. But please do not use before and after as accurate measures of the success of the platform. Obviously you can see massive issues and changes before/after but you can’t declare a winner. Besides, if the old platform was better, are you going to switch back? If the answer is no, then why bother caring? Just iterate and improve the one you are now on and move forward.
Outside of the major things like replatforms, I cannot express how important it is for you to test your thinking, even if it is as simple as the color of the Add to Cart button. Not to digress but my ATC button was red. I was told that was crazy, red is ‘danger’ and I should make it green, green means go. So I A/B tested it as I didn’t believe it mattered. Turns out it didn’t matter but I still made it green. Why? Because I was tired of hearing how dumb red was and the difference was zero, why argue over something of zero value?
