We often talk about the need to learn the process of science, rather than simply memorizing the things that science has discovered. In fact, “science literacy” is defined by most as a combined knowledge of process and information. Indeed, the national science education standards state that “A literate citizen should be able to evaluate the quality of scientific information on the basis of its source and the methods used to generate it.”
Evaluating the quality of scientific information is not easy, especially in areas in which the methods are complex. In my own field of psychology, most undergraduates are terrified of the core requirements for the major: intermediate statistics and research methods. These are notoriously difficult courses, but without them it is nearly impossible to evaluate the quality of any study in the field.
So if college students have a tough time understanding the scientific method or how to use knowledge of it to properly evaluate claims, how can we expect middle school students to learn this?
One solution is to teach the rather specific skill of skeptical thinking—separating science from pseudoscience. Because this approach is critical by nature, it promotes scientific thought in general. When we look for signs that something is pseudoscientific, we think about what methods would make it scientific. When we consider what questions need to be answered, we figure out how to answer them. For example, if someone claimed that studies showed that a bracelet had special powers to increase athletic ability, what would you want to know about how that study was conducted? The list (which might include how many people tested it, if all wore the bracelet, etc.) is one that tells you how to design a good study.
There are many colleges and universities with courses teaching about pseudoscience, but primary and secondary schools rarely have room in the curriculum. Supplementary programs are becoming more common and there are good indications that these programs are effective.
David J. Van Dyke received a JREF Educator Grant to execute such a program. He held a two-week workshop during which he taught a group of 8th Graders about the difference between science and pseudoscience using examples of claims from skeptic history, such as “The Mars Effect” and “Orgone Energy”.
Dr. Van Dyke is no stranger to this approach. His dissertation, available for download here, examined the relationship between origin beliefs (e.g., special creation) and science achievement. The good news is that beliefs do not seem to be a barrier to learning science for high school students, but they need the instruction to get there. Supplemental instruction in pseudoscience is an excellent way to increase science literacy.
The students in this year’s program recreated Emily Rosa’s “therapeutic touch” experiment, then chose a topic and designed their own double-blind study to test a claim. They presented their findings at a local science fair in a special category.
“Theraputic Touch” experiment
But how do we know that they learned anything? We know because Dr. Van Dyke measured it.
The students took a pretest and a post-test to assess learning. On average, student scores increase by 6.7 points out of 20—increases ranged from one to 14 points and none of the students’ scores went down.
This is often the best that we can do in real-world situations, but it leaves us with some questions. Any gains in scores could easily be attributed to something called “testing effects”—on average, most people perform better on a test the second time through it simply because they have experience with the test. This is why we need to compare treated subjects with those who have not received the treatment (control subjects), and he did. The average score of students in the control group remained the same, so even testing effects were not present. For the statistics-minded among you, a t-test comparing the change scores produced a Cohen’s d of 2.3 and a p-value of less than .001.
Furthermore, gender was not a factor in either performance on the pretest or the effects of the course. Both groups were ethnically and economically diverse.
In other words, instruction clearly mattered. The program works.
This was a volunteer program. The kids were not graded on participation and were not required to participate, so why did they? I think there are two things that make this program work. First, the students were rewarded for completion of the program with a book (Bausell’s “Snake Oil Science”). Second, and perhaps more important, they presented their findings at a local science fair. The opportunity to show off one’s work and compete for recognition (the fair was judged) is a strong motivator. It makes the work exciting and fun.
The JREF is proud to have played a role in the work Dr. Van Dyke has done and is equally proud to offer other educators the opportunity to conduct their own study sessions by providing the basic materials he used. We have edited the lesson plan slightly to ensure that it is flexible enough for many situations, but otherwise the materials are intact.
The JREF offers a limited number of grants to educators, including scholarships to attend The Amaz!ng Meeting. All educators can apply for the TAM Teacher Grant here. If you are an educator with a project you would like to implement, watch Swift (at randi.org) for announcements regarding the next round of awards.
If you would like to help us create more of these success stories and invest in our most precious resource, our teachers, a gift to the James Randi Educational Foundation is tax-deductible. You can also contribute directly to the TAM Teacher Grant, which provides registration to The Amaz!ng Meeting to educators here.