How to improve students’ critical thinking about scientific evidence

Introductory lab courses are ubiquitous in science education, but there has been little evidence of how or whether they contribute to learning. They are often seen as primarily “cookbook” exercises in which students simply follow instructions to confirm results given in their textbooks, while learning little.

In a study published today in the Proceedings of the National Academy of Sciences, scientists from Stanford and the University of British Columbia show that guiding students to autonomous, iterative decision-making while carrying out common physics lab course experiments can significantly improve students’ critical thinking skills.

In the multi-year, ongoing study, the researchers followed first-year students in co-author Douglas Bonn’s introductory physics lab course at the University of British Columbia. They first established what students were, and were not, learning following the conventional instructional approach, and then systematically modified the instructions of some lab experiments to change how students think about data and their implications.

One of the first experiments the researchers tackled involved swinging a pendulum and using a stopwatch to time the period between two angles of amplitude. Students conducting the traditional experiment would collect the data, compare them to the equation in the textbook, chalk up any discrepancies to mistakes and move along.

In the modified course, the students were instructed to make decisions based on the comparison. First, what should they do to improve the quality of their data, and then, how could they better test or explain the comparison between data and the textbook result? These are basic steps in all scientific research.

Students chose improvements such as conducting more trials to reduce standard error, marking the floor to be more precise in measuring the angle, or simply putting the team member with the best trigger finger in charge of the stopwatch.

As their data improved, so did their understanding of the processes at work, as well as their confidence in their information and its ability to test predicted results.

“By actually taking good data, they can reveal that there’s this approximation in the equation that they learn in the text book, and they learn new physics by this process,” said Natasha Holmes, the lead author on the study, who began the research as a doctoral candidate at UBC and is building upon it as a postdoctoral research fellow at Stanford.

“By iterating, making changes and learning about experimental design in a more deliberate way, they come out with a richer experience.”

Researchers found that students taking an iterative decision-making approach to the experiment were 12 times more likely to think of and employ ways to improve their data than the students with the traditional instruction. Similarly, the experimental group was four times more likely to identify and explain the limits of their predictive model based on their data.

Even more encouraging, these students were still applying these same critical thinking skills a year later in another physics course.

“This is sort of a radical way to think about teaching, having students practice the thinking skills you want them to develop, but in another way it’s obvious common sense,” said co-author Carl Wieman, a professor of physics and of education at Stanford. “Natasha has shown here how powerful that approach can be.”

The ability to make decisions based on data is becoming increasingly important in public policy decisions, Wieman said, and understanding that any real data have a degree of uncertainty, and knowing how to arrive at meaningful conclusions in the face of that uncertainty, is essential. The iterative teaching method better prepares students for that reality.

“Students leave this class with fundamentally different ideas about interpretation of data and testing against model predictions, whether it’s about climate change or vaccine safety or swinging pendulums,” Wieman said.

At Stanford, Holmes is expanding her research, applying these lessons to a range of undergraduate courses at different levels and subjects.

If iterative design can get first-year students to employ expert-like behaviors, the gains could be greater in advanced courses, she said. When students embark on an independent project, for instance, they’ll be much better prepared to face and clear any hurdles.

“Students tell me that it helped them learn what it means to do science, and helped to see themselves as scientists and critical thinkers,” Holmes said. “I think it’s done a whole lot for their motivation and attitudes and beliefs about what they’re capable of. So at least from that perspective, I think experiment design that encourages iterative thinking will have huge benefits for students in the long run.”