I became interested in using “big data” for A/B testing after a speaker from RedHat gave a talk to our area about it a couple of years ago. It’s a tantalizing idea: come up with a change, send it out on some small percent of your users, and pull it back immediately if it doesn’t work or isn’t better than the original. Even more amazing when you consider a “small percent” can be thousands and thousands of people – a dream for any researcher. Certainly, this connects to last year’s news on the controversy over Facebook’s A/B testing adventures.
The only con I can think of is that if something works or doesn’t work, you may not know why. We are always fumbling toward success, but maybe it’s not good to encourage fumbling over development of theory.
NPR’s Planet Money did a great show recently on A/B testing their podcast and the surprising results. They were also willing to think further about how it could be taken to an extreme, audience testing every segment of the show. Certainly worth a listen.