Implementing productive failure in a course on algorithms and data structures
We chose to implement productive failure, an innovative learning paradigm, in a course on algorithms and data structures.
We interviewed Tanmay Sinha, an expert on productive failure and a researcher in Professor Manu Kapur's group at D-GESS.“
„
“
- Sverrir & Felix:
- How is the PF instructional design distinct from other previously studied problem-solvingbefore-instruction strategies?
- Tanmay Sinha:
- Not all problem-solving-before-instruction (PS-I) designs are equivalent to PF, but only those that follow the design principles of PF as articulated in Kapur & Bielaczyc (2012).
These principles spanning the design of the problem, participation structure, and social surround can be operationalized into six concrete criteria: a) whether problems are designed to afford multiple representations and solution methods, b) whether affective draw has been considered in the problem design to stimulate students’ situational interest, c) whether group work (collaborative learning) is used during the initial problem-solving phase, d) whether the follow-up instruction phase builds on
failed/suboptimal student-generated solutions, e) whether the learning design emphasizes appropriate socio-mathematical norms (e.g., safe space to explore ideas), and finally, f) whether dialoguedominant discourse style is used to facilitate the instruction phase, as opposed to a one-way transmission of canonical information.
In fact, in a recent meta-analysis comparing PS-I with instruction-first approaches (Sinha & Kapur, 2020a, under revision), we coded these criteria for N = 166 comparisons (n = 12047 students) in the literature and discovered an average effect size of Hedge’s g 0.36 [95% CI 0.20; 0.51] favoring PS-I on learning outcomes of conceptual understanding and transfer. Crucially though, the effect sizes were even higher when PS-I was implemented with high fidelity to the design principles of PF, with effect sizes going up to Hedge’s g 0.58. Under assumptions of a normal distribution, this reflects up to a
64% likelihood that a random student picked from the PS-I condition (with high design fidelity to PF) will do better than someone from an instruction-first condition. be operationalized into six concrete criteria: a) whether problems are designed to afford multiple representations and solution methods, b) whether affective draw has been considered in the problem design to stimulate students’ situational interest, c) whether group work (collaborative learning) is used during the initial problem-solving phase, d) whether the follow-up instruction phase builds on failed/suboptimal student-generated solutions, e) whether the learning design emphasizes appropriate socio-mathematical norms (e.g., safe space to explore ideas), and finally, f) whether dialoguedominant discourse style is used to facilitate the instruction phase, as opposed to a one-way transmission of canonical information. In fact, in a recent meta-analysis comparing PS-I with instruction-first approaches (Sinha & Kapur, 2020a, under revision), we coded these criteria for N = 166 comparisons (n = 12047 students) in the literature and discovered an average effect size of Hedge’s g 0.36 [95% CI 0.20; 0.51] favoring PS-I on learning outcomes of conceptual understanding and transfer. Crucially though, the effect sizes were even higher when PS-I was implemented with high fidelity to the design principles of PF, with effect sizes going up to Hedge’s g 0.58. Under assumptions of a normal distribution, this reflects up to a 64% likelihood that a random student picked from the PS-I condition (with high design fidelity to PF) will do better than someone from an instruction-first condition. - Sverrir & Felix:
- Is PF appropriate for every subject type and student age group? Do some domains seem to benefit more from PF than others?
- Tanmay Sinha:
- Results from the aforementioned meta-analysis suggest that on average, PS-I with high design fidelity to PF is a promising instructional approach to facilitate students’ conceptual understanding and transfer. However, it is indeed important to acknowledge the variation in effects.
Relative to instruction-first approaches, effects favoring PS-I are higher if students belong to higher grade level, and for longer interventions spanning a few days. However, both these factors are confounded to varying degrees with the extent to which PS-I follows PF design criteria. PS-I interventions with 6th – 10th graders and undergraduate student populations have higher fidelity of implementation to PF, which might partially explain the increase in effect size relative to younger student populations (e.g., 2nd – 5th graders). Further, if educators wish to implement PS-I for shorter interventions (that is, those spanning a few hours), cramming too many PF design features may not be optimal. For shorter interventions, results from the meta-analysis suggest that affective draw of the problem design is important to consider and using individual work as the participation structure in the problem-solving phase is better than using group work.
Relative to instruction-first approaches, effects favoring PS-I are also higher if the intervention targets domain-specific STEM concepts (e.g., those in mathematics, physics, chemistry, biology, medicine, environmental science), as opposed to domain-general skills (e.g., control of variables strategy, water jug problems, Rubik’s cube). For domain-specific STEM concepts, effect sizes favoring PS-I over instruction-first approaches range from Hedge’s g 0.24 to 0.56. For domain-general skills, current scientific evidence suggests that instruction-first approaches are better.
- Sverrir & Felix:
- What are examples of recent pedagogical innovations within the paradigm of PF at the Professorship for Learning Sciences and Higher Education, DGESS, ETH Zürich?
- Tanmay Sinha:
- We are working on two major strands of research to advance the science of learning from failure and implications for pedagogical practice.
First, we have begun investigating deliberate, guided failure as a novel scaffolding strategy within PS-I. As opposed to the classic implementation of PF where students are not given any explicit scaffolds to nudge their problem-solving towards failure (Kapur & Bielaczyc, 2012), we are currently evaluating the pedagogical value of deliberately nudging students to generate specific suboptimal representations and solution methods (e.g., using a bar chart or one-dimensional histogram instead of a scatterplot, when reasoning with a bivariate dataset). Consistent experimental results from a classroom study (Sinha, Kapur, West, Catasta, Hauswirth, & Trninic, 2020) and a follow-up lab study (Sinha & Kapur, 2020b, under revision) suggest that deliberate, guided failure facilitates students’ learning at the posttest, especially for questions requiring adaptations of learned procedures (e.g., non-isomorphic conceptual understanding and transfer questions). Further, students exposed to failure-driven scaffolding during the PS-I intervention show a higher reasoning quality at the posttest, that is, a higher proportion of complete mathematical or non-mathematical elaborations. These learning effects are more pronounced relative to success-driven scaffolding and a classic PF condition. Deliberately guiding students towards failure during the initial problem-solving phase of PS-I affords exploration of the problem-space, provides variable practice, and traps potential misconceptions (that arise from students making a misprediction). Despite challenging students’ understanding (and making them uncomfortable in the short-term), deliberate, guided failure improves students’ readiness to learn from the follow-up lecture (plausibly more so than success-driven scaffolding and classic PF).
Second, we have begun expanding the explanatory basis of PS-I by consolidating evidence for the impact of several retrospective and process measures during our study contexts. For instance, in the aforementioned experimental interventions, we measure specific learning mechanisms that might explain the differential advantages of deliberate, guided failure over success-driven scaffolding and classic PF. Additionally, we also conduct multimodal learning analyses using students’ facial expressions from fine-grained video recordings for inferring the incidence and dynamics of several emotions as they naturally arise during the preparatory problem-solving (Sinha, 2020). Results suggest that after controlling for students’ incoming cognitive and motivational profile (e.g., their prior knowledge, learning goal orientation), explicitly scaffolding preparatory problem-solving activities towards failure: a) facilitates students’ knowledge gap awareness, or the realization of what is known and not known about the targeted concept, b) triggers state curiosity, or students’ desire to know more about the canonical solution to fill these knowledge gaps, c) triggers cognitive dissonance, or an aversion/uncomfortable tension during the problem-solving process, d) increases germane cognitive load, or the mental effort devoted for processing/constructing relevant task information and identifying deep features, e) induces negative (as well as positive) affect, or subjective experiences of pleasant and unpleasant emotions, and f) eliminates under- and- overconfidence biases and facilitates metacognitive calibration, or accurate self-evaluation of problem-solving performance. Further, preparatory problem-solving (especially in the presence of failure-driven scaffolding) is an intense emotional experience, where students experience an abundance of unconventional negative emotions that differentially impact learning (e.g., shame, anger, and disgust are positively correlated to posttest scores, while contempt and pain are negatively correlated). Further, these results endorse previously established work on the presence and positive influence of emotions like surprise and interest in generative problem-solving. The psychological functions of emotions (backed up with retrospective self-reports) offer plausible explanations for the impact on learning.
- Sverrir & Felix:
- Some students in our course reported frustration when attempting the PF tasks that we gave them. How can instructors help students manage frustration and other negative emotions when solving novel and challenging problems? In such a scenario, is it even ethical to use deliberate, guided failure as a scaffolding strategy?
- Tanmay Sinha:
- We believe that the extent to which implementation of a failure-first instructional approach is an ethical scaffolding strategy, depends largely on a) how learning activities foregrounding failure are framed, and b) whether there exists a supportive social surround (positive and trustworthy climate) in the classroom to facilitate appreciation for failure-driven learning. There are several implications following from our research to potentially address the negative effects of engaging in failure-driven problem-solving prior to instruction.
First, setting the stage is important. As outlined in Kapur & Bielaczyc (2012) and Sinha & Kapur (2020a, under revision), it is critical for educators to a) emphasize the problem-solving phase of PS-I as a safe space to explore and generate ideas, b) enforce appropriate social and mathematical norms around the activity (e.g., it is okay not to be able to solve problems as long you try various ways of solving them; highlighting to students that there are multiple solution approaches for the task; setting the expectation that understanding why and under what conditions some solutions are better than others is important), and c) provide affective/motivational support to facilitate persistence, or keep students engaged in generation. By adopting these strategies, the negative influence of threats to students’ status and respect is mitigated. It is entirely legitimate to expect such threats to arise, for instance, from factors including but not limited to coping with uncertainty in solution generation (because of lack of verifiable outcomes), task conflict with peers, etc.
Second, the implementation of deliberate, guide failure as a scaffolding strategy during preparatory problem-solving should go hand-in-hand with fostering positive teacher-student relationships (Jennings & Greenberg, 2009). For instance, students are more likely to trust the pedagogical value of classroom activities, if they have a good interpersonal rapport with the teacher, and trust the teacher to show genuine interest in (and meet) their developmental, emotional, and academic needs. For teachers, therefore, it is equally critical to cultivate positive relationships in the classroom as well as adopt a pedagogical value-style framing of failure-driven scaffolding activities (that clearly emphasizes the utility of engaging in such activities), to make students increasingly more comfortable with the uncomfortable. Along with providing empathy concerning possible frustration of students, a positive classroom climate is also likely to increase their awareness of the pedagogical benefits of deliberately engaging in errorful generation, given that students generally undervalue and underutilize doing so (Pan, Sana, Samani, Cooke, & Kim, 2020).
Third, educators can also devote classroom resources to help students understand and manage negative emotions that arise during the problem-solving phase of PS-I (see for example, the RULER curriculum, (Brackett, Bailey, Hoffmann, & Simmons, 2019) developed to help students of all ages recognize, understand, label, express, and regulate their emotions). Recent meta-analytic evidence (MacCann, Jiang, Brown, Double, Bucich, & Minbashian, 2020) suggests that knowledge about the causes and consequences of emotions, and knowing how to manage emotional situations are core aspects of emotional intelligence that predict academic performance. Our results (Sinha, 2020) also suggest that experiencing moderate levels of negative emotions are okay as long as they are appraised appropriately, and actions are taken to move forward in the task to overcome the discomfort caused due to such emotions. Condemning negative emotions totally may therefore obstruct practicing emotional intelligence. However, in extreme cases of disengagement (e.g., when a student completely withdraws from the problem-solving task), educators can also choose to implement strategies to reduce the effects of certain negative emotions (see Lerner, Li, Valdesolo, & Kassam, 2015 for a review).
- Sverrir & Felix:
- Are there any common mistakes that instructors make when constructing their own PF sessions? If so, can instructors make use of some educational technology to avoid such mistakes?
- Tanmay Sinha:
- Exposing students to PF (and PS-I more generally), can take many different forms, some of which are effective, and some of which are not. In particular, when conducting PF sessions, providing students with standard (textbook) practice problems as a way of preparation for learning from a lecture is a common fallacy. Instead, preparatory problems should be chosen such that they admit multiple representations and solutions and use variant-invariant features (e.g., datasets with similar mean and standard deviations, but contrasting visualizations), contrasting cases (e.g., comparing similarities and differences across multiple datasets to rank order based on criterion), etc to create opportunities for failure in problem-solving. Further, when conducting PF sessions, a common fallacious view is to only focus on why right is right (by providing students with canonical information in a clear and well-structured manner) during the instruction phase. Instead, it would be imperative to adapt the instruction slightly to also foreground how typical (common) suboptimal student solutions relate to the canonical information and emphasize why wrong is wrong. Such a compare and contrast style instruction will help students in concretely identifying discrepancies in their reasoning. Finally, a third fallacy when conducting PF sessions might be the failure (or, lack of emphasis) to maintain a supportive social surround throughout the problem-solving and instruction phases. Instead, it would be fruitful to situate problem-solving practices in a classroom climate that welcomes failures as important learning opportunities.
Educational technologies (and research advances in artificial intelligence in education, more generally) offer a promising solution to mitigate or reduce the impact of the aforementioned fallacies when designing PF sessions. The possibilities are enormous, but here we provide a few examples. For instance, designing contrasting cases on-the-fly to problematize student reasoning during preparatory problem-solving is one relevant direction. For a large classroom setting, automated grading with educational technologies and insights from summarization research in natural language processing also hold potential in informing the instructor about typical suboptimal solutions from students prior to the delivery of the lecture. Finally, to enforce norms around problem-solving activities and provide students affective support for persistence, implementation of pedagogical computer agents (Johnson & Lester, 2016) that can stay with students during the process of problem-solving and adapt a conversational-style of interaction, is also a potentially worthwhile use of educational technologies in the service of reducing teachers’ classroom management load.
- Sverrir & Felix:
- During the COVID-19 crisis, instruction has largely moved to an online format, a change that may end up being permanent in some capacity at some institutions. How can instructors conduct PF session effectively under a remote setting?
- Tanmay Sinha:
- As a teacher facilitating remote learning designed on PF principles, it is critical to constantly reassure students that the goals of preparatory problem-solving are not necessarily to reach a correct solution, but instead to merely explore and evaluate the affordances of different problem-solving strategies based on what students already know. With a lack of physical presence, it is also important to motivate students to keep generating multiple solutions to the given problem, and sharing snapshots of their individual or group work at regular intervals (using online learning tools, e.g., screen sharing, breakout rooms) so that progress can be monitored. When designing the lecture for online delivery or via a pre-recorded video, it becomes even more important to carefully consider design principles drawn from research in multimedia learning (Clark & Meyer, 2016), so as not to overwhelm students with information.
- Tanmay Sinha:
- References:
Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. Journal of the Learning Sciences, 21(1), 45-83. https://doi.org/10.1080/10508406.2011.591717
Sinha, T., & Kapur, M. (2020a). When problem-solving followed by instruction works: evidence for productive failure. Manuscript under revision
Sinha, T., Kapur, M., West, R., Catasta, M., Hauswirth, M., & Trninic, D. (2020). Differential benefits of explicit failure-driven and success-driven scaffolding in problem-solving prior to instruction. Journal of Educational Psychology. Advance online publication. https://doi.org/10.1037/edu0000483
Sinha, T., & Kapur, M. (2020b). Robust effects of the efficacy of explicit failure-driven scaffolding in problem-solving prior to instruction: A replication and extension. Manuscript under revision
