Last week, I published a post talking about data manipulation and the importance of controls during an experiment. It turns out, this is a hugely complex subject, so I wanted to follow that whole train wreck with something a little simpler.
Here are a few more examples of scientific method. My hope is that these will illustrate how important it is that you use controls, and that you be wary of scientists who don’t.
The Toaster Experiment. Khan Academy has a fantastic introduction to scientific method. I highly recommend you read through their examples. In this one — “failure to toast” — they outline a familiar household problem in which we use basic scientific method without even realizing it.
- Observation: My toaster won’t toast!
- Question: Why won’t my toaster toast?
- Hypothesis: My toaster won’t toast because the outlet is broken.
- Prediction: If I plug the toaster into another outlet, it will toast.
- Test: Plug the toaster into another outlet. If it toasts, the first outlet was broken. If it doesn’t, something else is wrong.
Now, for the control. What if we plug the toaster into another outlet, but it doesn’t work? Should we assume that the toaster is broken? Perhaps. But what if the toaster is actually fine, but the second outlet is also broken? We might accidentally assume the toaster’s broken when it’s not.
There’s a simple way to get around this uncertainty:
- Include a control. Plug a working lamp into the second outlet. If the lamp turns on but the toaster doesn’t, you’ve got yourself a broken toaster. If the lamp doesn’t turn on, you might need to reset that breaker.
Watering a pot of seeds. Read the next page on Khan Academy to get another great example of a controlled experiment. Here’s a quick summary:
- Observation: My pot of seeds didn’t sprout!
- Question: Why didn’t my seeds sprout?
- Hypothesis: My seeds didn’t sprout because I forgot to water them.
- Prediction: If I remember to water my seeds, they will sprout.
- Test: Plant a new pot of seeds, but this time, water it.
Of course, if these seeds grow, you can probably assume that it’s because you watered them. But wait — maybe there was something wrong with the last batch of seeds! So how do you know for sure?
You guessed it:
- Include a control. Plant another pot of seeds exactly the same way, but do NOT water this one. If the watered pot sprouts and this one doesn’t, you’ve found the problem. If it sprouts even without water, you know something was wrong with your last batch of seeds.
This example also does a great job illustrating the importance of sample size. If you plant 10 seeds and 9 of them sprout, you can pretty safely call that a successful outcome. However, what if you only planted 1 seed in the watered pot? If it doesn’t sprout, we would make the mistake of assuming the watering didn’t work. In reality, that seed was probably just a late bloomer. This would give us what’s called a false negative. Similarly, if we only planted 1 particularly tough seed in the un-watered pot, and it miraculously sprouted, we would assume (wrongly) that the watering isn’t necessary. That’s a false positive. More on these later!
I’ll end with a real-life example of an experiment that did not use controls. For this, I’ll turn to my favorite go-to example of terrible science: Andrew Wakefield and his “vaccines cause autism” paper.
You may already know that in 1998, Wakefield conducted a study to see whether there was a cause-and-effect relationship between MMR vaccines and developmental delays in kids. He fudged the results (and did a lot of other sketchy shit), made wild claims that the MMR vaccine causes autism and intestinal issues, and started an anti-vaccine movement the effects of which we can still feel today.
I mentioned in my last post that for a study like this, researchers will try to get groups of hundreds of people at least… often thousands. Wakefield used only 12. So, already, the sample size is a giant red flag.
Now let’s talk controls. Here’s a basic example of how this study might be carried out, if proper scientific method were followed:
- Observation: Some children are manifesting signs of behavioral issues like autism.
- Question: Why are these kids showing behavioral issues?
- Hypothesis: The kids are showing behavioral issues because they received the MMR vaccine.
- Prediction: Children who received the MMR vaccine show significantly more behavioral issues than children who did not get the vaccine.
- Test: Measure rates of autism in a group of children who received the MMR vaccine.
- Include controls: Compare autism rates in a group of children who did not receive the MMR vaccine.
Basically, Wakefield fudged his way through everything after #5. He took 12 kids with developmental issues and looked back to see whether they’d gotten the MMR vaccine. Turns out, yep! They had!
But think. In 1998, 600,000 children per year were getting the vaccine in the UK. Autism and other developmental disorders tend to become obvious when the child is, you know, developing. That means that, by nature, kids will be getting their shots around the same time that they start showing signs of a disorder. Does that mean the shot is causing the disorder?
No. That would be like saying puberty is caused by internet porn searches.
So what would you do with this information? You, who by now understands the importance of controls, might go find a ton of kids without developmental issues, and see how many of them had gotten the vaccine. Turns out, it’s a lot! In a 2002 study in the New England Journal of Medicine, a group of 440,655 vaccinated kids was compared to a group of 96,648 unvaccinated kids. That’s a hell of a sample size. There was no difference in the rates of autism between the two groups.
Importantly, those researchers controlled for confounding variables. They took care to make the two groups as similar as possible. This means the only difference was that one group got the MMR shot, and the other did not.
There were many other issues with Dr. Sketchy McDodgyface’s paper; Vox summarized these very nicely earlier this year, and I’m sure we’ll touch on these in later posts. Perhaps the most colorful snippet is that the guy paid his 10-year-old kid’s friends to donate their blood for the study. That doesn’t have much to do with controls… I just really like telling people about it.