credit: elisbrown

Everyone Experiments

Scientists aren’t the only ones who do experiments.  If you’ve ever tried to optimize a cake recipe, bought Ad Words to market your business, or tried everything you could think of to help your baby sleep, you’ve done an experiment.

The key idea here is that you were trying to optimize something (cake density, conversion rates, or baby happiness) by varying something else (amount of baking soda, the Ad Words you chose, to bounce or rock?)

Take a moment to think about it: what have you tried to improve, optimize, or experiment on lately? We’ll wait until you come up with something…..

By the end of this post, you will know about a technique you’ve probably never heard of to get better experimental results, with fewer tests, in much less time.  Sound good?

Ways to Play

Of course, there are lots of different ways to do an experiment.  Since pretty much everyone who was alive during the 1980′s has played the board game “Battleship,” we’ll use that as our analogy to experimentation.

It’s an apt comparison: when you experiment, you’re searching for an improved or optimum response (This is your readout, like a tastier cake).  You may only be worried about how much baking soda to add (one dimensional) or you may be looking at the interaction of baking soda and oven temperature (two dimensional).

On the battle ship board, you’ve got two dimensions to search in.  Think about the numbers 1-10 down the side as the amount of baking soda in your cake and the letters A-J across the top representing your oven temperatures.  The perfect cake recipe lies where the cruiser ship is hidden – a winning combination of both the perfect amount of baking soda and the perfect temperature.

How do you find that winning combination?  Well, you could start at the top, making a big batch of cake batter with 1 unit of baking soda and cooking it at the lowest oven temperature.  Then, you’d taste it, turn up the temperature 10 degrees, and bake another batch with just 1 unit of baking soda.  Once you had gone through the entire temperature range, you’d make some batter with 2 units of baking soda, cool the oven down and start over until you had gone through 100 cakes!

A great way to lose a game of Battleship - or your sanity.

No one is that patient (or dumb)

Seriously, though, I don’t know very many people who would do this (aside from Cook’s Illustrated, which I happen to love…)  Most of us playing Battleship or optimizing a recipe would try to sample across the design space, trying various combinations, but certainly not EVERY combination.

How many of you played battle ship like this?

Now we're getting somewhere!

You are guaranteed to sink that sub (or find the best cake recipe) in 36 moves, about a third of what the “thorough” method would’ve taken.  Don’t you feel smart?

The Big Secret

In the immortal words of Ron Popeil, “But wait, there’s more!”  It turns out there’s an even BETTER way to do this experiment, it just takes a little math.

The technique I’m talking about is actually called “Design of Experiments” or DOE to those in the know.  Why is it a secret?  I have no idea, but I managed to make it through five and a half years of a PhD program without hearing the phrase even once, and I am not alone.  Most of the biomedical scientists I know, even at the faculty level, have never heard of DOE.

Here’s why that’s sad: DOE is like a heat-seeking missile on the Battleship board.  Imagine if every time you made a move, I told you just how far away you were from the cruiser.  It might look something like this:

You cannot hide from me, Mr. Cruiser!

You’d start this DOE enabled game by sampling from the various columns and rows (2 units of baking soda and a 410 degree oven, then 9 units of baking soda and a 350 degree oven).  It wouldn’t take you long to home in on the right mix:

That's one delicious cruiser cake.

And that’s the beauty of DOE – you can test multiple factors (like oven temp or baking soda) across multiple levels (from one unit, to ten units of baking soda) all at the same time. The design is optimized to give you the biggest bang for your buck (i.e. to minimize the number of experimental runs you need to find the right mix.)

This is extremely helpful for a couple of reasons.  First, your factors might have interactions that you’d never discover if you tested each one on its own. For example, the temperature of the oven could have an effect on how well the baking soda works, so if you tried to find the best amount of baking soda at 350 degrees, and THEN used that static amount of soda to find the best oven temperature, you might come up with a sub-optimal cake because you’d never realize that you need more baking soda in the hotter oven.

The other helpful part of DOE comes when you’re thinking about more than two factors. What if you wanted to also test which size pan – 8 inch or 9 inch – gave the best cake?  Well, now you’ve got a three dimensional space to search in.

Adding 2 pan sizes gives you 200 combinations to test using the "brute force" approach.

Or what if you’ve got 2 pans, each with three different finishes (dark, light, and non-stick), and you want to check 10 butter temperatures and 5 flour ratios along with your baking soda and oven temp ranges.  I won’t even try to illustrate the 6-dimensional space of your new experiment, but it would create 30,000 possible combinations for cake.  DOE can point you to the optimum answer (with interactions!) in as few as 28 runs.

I don’t know about you, but I’d rather make 30 cakes than 30,000 cakes.  Sounds like a diabetic coma waiting to happen.

Hopefully you can see the value and importance of DOE in your own work or life. If you’re interested in learning more, check out how a middle school student used DOE to build a catapult, read about optimal design on Wikipedia, or check out a tutorial on using JMP for custom designs.

Or, let me know in the comments if you’d like to learn more. I know a few people who do DOE for a living, and they may be willing to share some of their own tips and tricks for getting started.

No related posts.