Revealing Students’ Computational Thinking in Problem-Solving Using Cognitive Interviews and Think-Aloud Protocols
July 17, 2019 Feiya Luo
Author: Feiya Luo
Contributors (names in alphabetical order): John Hampton, Maya Israel, Todd Lash, Ruohan Liu, Michael McKelvey, Wei Yan.
This Spring semester, we piloted a series of Action Fractions lessons with 4th-grade students. During the lessons, students built projects using computational thinking (CT) skills such as decomposition, sequencing, and repetition to solve math projects focused on fractions. As part of the effort to understand students’ thinking during problem solving, we conducted “cognitive interviews,” also known as the think-aloud protocol (TAP), to solicit students’ immediate thoughts and reactions during problem solving. In our case, the purposes were to a) have students verbalize their thinking as they solved the CT+fractions problems, b) identify learning gaps and pitfalls in regard to CT understanding, and c) help us validate the assessment items–i.e. to see if there is any misalignment between students’ actual problem-solving and the item developers’ intention.
The assessment items used in the cognitive interviews were selected from the CT assessments designed to assess student learning after they go through the Action Fractions lessons. To ensure meaningful data collection, the items included multiple-choice, open-ended, and fill-in-the-blank questions that covered all important CT concepts present in the lessons. From a data collection perspective, no True or False questions were selected, since they would not provide as much rich data compared to the other question formats used.
Thirteen students participated in the cognitive interview. Each participant was presented with 3 items, which took them an average of 10-15 minutes to complete the items while verbalizing and sharing their thoughts. Each interview started with a brief introduction of what the participants were expected to do and the interviewer modeling what a fourth-grade fraction problem “think-aloud” looked like. With a total of 37 item responses, intriguing patterns emerged. The following are some highlights:
Sequencing turned out to be the concept that students were most comfortable with. Almost all responses to the sequence items (see Fig. 1 for example) were correct and students presented problem-solving rationale intended by the items.
Items tapping into conditional logic (e.g., if…then statements) revealed that while participants may understand this concept in daily life contexts, such as “If it is cold, then put on a jacket,” they did not know how to evaluate the condition in computing. For example, when asked to tell what would happen when “If (7<5), sound pop; else, sound bing” (see Fig. 2), none of the three participants answered it correctly. The practical implication was that when teaching conditional logic, it is important to explicitly explain that students have to first decide if a condition is True or False (in this case, decide that the condition “7<5” is false) before moving to the next line of code.
Decomposition items received mixed responses. For example, while the 20-second audio snippet shows how this participant correctly answered the following decomposition item (Fig. 3), another participant was lost over the fact that there were more than one unknown values in the equation.
Audio file transcript: Because someone already ran two. So, if we plus 3, that would equal 5. And, then add two more and that’d be 7. And, since it’s 7 miles and it’s a team, they would win because it equals up to 7.
The cognitive interview also revealed that the phrasing of items designed to assess CT in the context of math may need to be distinguished from typical math word problems. The reason is that, when students see a problem that involves math, they are inclined to directly solve the math instead of providing the steps (decomposing) to solve the problem. Therefore, distinctly-worded CT items may be more efficient in prompting students to use computational thinking during problem-solving.
Please note that the aforementioned assessment items are only in their “beta versions.” The findings of this study will be used by researchers on this project to revisit, redesign, and/or revise these items before large-scale administration in the following years. If you have any questions or concerns, please feel free to contact us for more details.
Acknowledgement: CTRL thanks Dr. Brian Douglas Gane (Visiting Research Assistant professor at UIC), Dr. James Pellegrino (Distinguished Professor at UIC), and Noor Elagha (doctoral student at UIC) for providing support to this study.