I worked at Booking.com several years ago. Booking.com is famous for their A/B testing culture. They test a lot. Probably too much.

While I was there, they were just starting to run A/B tests for their hotel-facing web site, and I had the amazing privelege of helping with some of this. In fact, I was even offered a promotion to become the tech lead for the newly-forming hotel-facing experiment team. I initially was excited to accept, because the work was fascinating, and I really enjoyed working with the data scientists in this problem space.

But there was a problem.

The work was really hard.

That wasn’t the problem, actually. That made it interesting.

The problem was that explaining this hard work to the product owners and other stakeholders who would be using the results of these hotel-facing A/B tests to make business decisions was hard. It was so hard, in fact, that most of the POs I was talking to would get frustrated and just ask for a simple explanation.

Of course I understand that desire. But sometimes there simply are not simple answers. In such a case, do you offer a simple-but-wrong answer, or a complex-but-right answer?

I concluded that, in that particular case, the better answer for these POs was no answer at all. It’s better to use your subjective intuition, than to rely on incorrect “objective” answers. Naturally, they weren’t interested in paying me for that, though, so I left the company. You can predict Booking.com didn’t agree with me, and they now have several teams dedicated to A/B testing. And last I heard, they were doing some amazing things with them, and no doubt, some POs are constantly misusing the results to make bad decisions.

I’ve had this same situation come up a few other times since then: Someone in management wants answers (via an A/B testing tool, or some other “hard statistics”) that simply isn’t possible to do reliably. Or the cost to do it reliably would far outweigh the value of the decision to be made.

The best answer can be no answer. But convincing people of this is often hard.

Have you been in this situation? How did you deal with it?