Testing with simulation

A recurring theme in this book is to use what's known as Monte Carlo methods to quantify the quality of our algorithm. Using this technique, we will time and again create groups of random data to see how well our algorithms (ignorant of how we created the data) perform at detecting the differences. In the previous chapter, most of the tests that we developed had deterministic test cases. In this chapter, the test cases will be specified by the parameters that we use, but all of the data that we create will be random.

First things first, we need to develop a small framework that we can use to run our simulations. In order to do this, we should more clearly articulate our scenario. Here's a more thorough definition.

Imagine we have a website, and we want to test three different treatments of the headline on it. We decide to run a test by randomly assigning every visitor to one of the three treatments. Our goal is to exploit our three treatments by figuring out which headline...