A/B testing is to product development as test driven development is to API design

I have had a lot of conversations about A/B testing that remind me of those that I had back in the days when Test Driven Development came on the scene.

It ain’t about tests

Much of the talk at the time was about testing and tests, focused on
fixing bugs and finding regressions. This is incredibly important,
especially as it was also growing in popularity at the same time that
dynamic languages were coming back en force (trust the tests not the
compiler!) as well as rich refactoring tools. The tests were a safety
net to allow you to say “yes” to evolving the code base versus a de
facto “no.”

I quickly found that the quality of my code was better with the right design because I was now thinking about the interface first, and the initial point of view was that of a client to the code base. End result: a much better design.

I also found that my tests were amazing documentation for the API. Screw pure text docs. Commenting the code is great but again favors the implementation. This way the core implementation docs were for the implementors and the tests had docs for the consumers.

Building product

Bringing it back to A/B testing, I have seen many talking about the effects that measuring your product has, and the effects of the backend of the process (running reports etc), but one of the most exciting side effects has been on the front loading of product development.

Recently someone told me that they didn’t have time to do A/B and “isn’t it a waste to build more than once?”

I have found that not only isn’t it a waste, but it is faster to build product with an A/B approach and the results are better.

I think that I am old enough to know that I am not always good at predicting the future, and when you are building a product you are predicting how your users will use your work. I believe in “instinct”and having knowledge of what to build, but I want to mix that with a healthy amount of data and trials.

I also believe that good ideas can come from anywhere, and any one on the team. If you have a “product manager” they aren’t the only holders of great ideas that the product should do.

I have been involved on projects that changed from one of these environments to the other:

Top Down

– team discusses what should be in the next release
– team disagrees on some points
– best debater may win, else
– the person who can raise it up the chain plays The Boss Card: “Well,
the Boss wants us to implement X so no more point in arguing.”

This can easily result in the team not feeling ownership and they end up more disgruntled.

Not only may the boss be wrong, but they may have different context….
And then there is even pure misunderstanding and Chinese whispers.

Team Ownership

– team builds out their hypothesis on what would result in hitting the
business outcomes
– instead of lengthy argument, build different versions and measure

Now the data wins versus personality, you learned something about you
product and users, and you iterate from there.

It is often quicker to build out a prototype or live feature vs. arguing about whether it will rain tomorrow. Acknowledge that users will surprise us, and then embrace A/B at the heart of your product development efforts and run like the wind.