Roelle and Berthold (2017) showed in a university-student experiment (N = 192) that the net benefit of implementing learning tasks in a closed-book (retrieval-based) rather than open-book style depends on the tasks' complexity: with adjunct questions over expository chemistry texts, closed-book implementation produced a larger net benefit for low-complexity summarization questions than for high-complexity inference-generation questions, where it also reduced the inferences learners produced. The authors left open whether the complexity-by-implementation interaction is robust once learners' anticipation of upcoming task demands is controlled, whether prior knowledge and verbal ability moderate it, whether the double-edged pattern generalizes to other constructive learning tasks, and how feedback or restudy opportunities after closed-book attempts rebalance retrieval benefits against performance costs. Later research shows that the generalization sub-question is now substantially answered: closed-book formats consistently curtail the execution of generative activities across concept mapping, self-explanation, teaching, essay writing, question generation, and drawing, whereas learning-outcome effects remain mixed, and closed-to-open switch formats are a promising optimization. The surviving open core comprises anticipation-controlled estimates of the interaction, learner-characteristic moderation of it, the task, implementation-quality, and learner factors that determine the direction and size of the outcome effect across constructive learning tasks, optimal access and feedback schedules, and an integrative account of when retrieval's consolidation function outweighs its costs to generative processing.

Background

Retrieval practice (the testing effect) is the finding that actively retrieving information from memory, as in practice testing, improves long-term retention more than restudying the same material. The effect is well established for relatively simple materials, but van Gog and Sweller (2015) argued that it decreases or even disappears as the complexity of the learning material increases, making complexity a central boundary condition of retrieval-based learning. In parallel, research on generative learning studies activities such as summarizing, concept mapping, self-explaining, and drawing, in which learners actively construct coherent mental representations of to-be-learned content instead of passively receiving it. The two strands intersect in the question of how a learning task that elicits generative activities should be implemented: in an open-book style, with access to the learning material while working on the task, or in a closed-book style that requires learners to first retrieve the relevant information from memory. Agarwal, Karpicke, Kang, Roediger, and McDermott (2008) found that closed-book and open-book tests supported comparable final retention; Karpicke and Blunt (2011) then found that free recall (pure retrieval) outperformed open-book concept mapping, sparking debate over whether generative tasks are enhanced or undermined by retrieval demands. Roelle, Roelle, and Berthold (2019) later reported inconsistent test-based-learning effects for higher-level versus lower-level questions, underscoring that complexity effects in this literature are far from settled. Roelle and Berthold (2017) brought the strands together in an experiment with 192 university students using adjunct questions, that is, questions provided together with expository chemistry texts. They orthogonally varied (a) question complexity: low-complexity questions required summarizing provided information, high-complexity questions required generating inferences from it; and (b) implementation style: closed-book questions, which require retrieval, versus open-book questions, which do not. On immediate and delayed criterion tests the net benefit of closed-book implementation was larger for low-complexity than for high-complexity questions, and closed-book implementation reduced the number of inferences learners generated. Retrieval can thus undercut exactly the generative processing that complex tasks are meant to elicit, a double-edged effect. The authors' discussion identified four unresolved issues that jointly define the boundary conditions of incorporating retrieval into learning tasks: (1) learners could anticipate the summarization demands after the first few texts, so the complexity-by-implementation interaction may be mis-estimated unless such anticipation effects are controlled; (2) individual learner characteristics, in particular prior knowledge and verbal ability, were unexplored as potential buffers against retrieval failure; (3) it was unknown whether the double-edged effects generalize to other constructive learning tasks that direct learners to specific knowledge-construction activities, such as concept mapping, self-explaining, or modeling; and (4) it was unknown how feedback or the opportunity to restudy after closed-book attempts changes the balance between retrieval's benefits and its performance costs.

Problem Statement

Roelle and Berthold (2017) asked under which boundary conditions incorporating retrieval into learning tasks — implementing the tasks in a closed-book style that requires learners to retrieve information from memory rather than an open-book style that leaves the material accessible — net-benefits learning, and how this incorporation can be optimized. Their experiment established that the net benefit of closed-book implementation is larger for low-complexity tasks (summarization) than for high-complexity tasks (inference generation), but left four linked questions open that together define the completion boundary of this problem. First, robustness to anticipation: does the complexity-by-implementation interaction persist when learners cannot anticipate upcoming task and summarization demands, so that the interaction is estimated without anticipation confounds? Second, learner-characteristic moderation: do individual learner characteristics, in particular prior knowledge and verbal ability, moderate the interaction, for instance by buffering learners against retrieval failure on high-complexity tasks and thereby attenuating the complexity-by-implementation effect? Third, generalization and its determinants: do the double-edged effects found with adjunct questions generalize to other constructive learning tasks that direct learners to specific knowledge-construction activities, such as concept mapping, self-explaining, or modeling? Later work (2014-2025) has largely established that the qualitative pattern — closed-book implementation curtails the execution of generative activities while outcome effects vary — occurs across such tasks, so the open part of this sub-question is which task, implementation-quality, and learner factors determine the direction and size of the outcome effect. Fourth, feedback and restudy: how do feedback on, and the opportunity to restudy or reinspect the learning material after, closed-book attempts alter the balance between retrieval's benefits and its performance costs — including whether fixed closed-to-open sequences, learner-controlled switching, or externally scheduled access phases are superior, and for whom? These boundary conditions — anticipation-controlled estimates of the complexity-by-implementation interaction, tests of moderation by prior knowledge and verbal ability, the task, implementation-quality, and learner factors that determine the direction and size of the outcome effect across constructive learning tasks, and comparisons of access and feedback schedules — remain the target of the problem, along with the experiments that would settle whatever the available evidence leaves unresolved and an integrative account of when the consolidation function of retrieval outweighs its costs to the construction of coherent mental representations.

References

  1. Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: The complexity of the tasks matters. Learning and Instruction, 49, 142-156. https://doi.org/10.1016/j.learninstruc.2017.01.008
  2. Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L., III, & McDermott, K. B. (2008). Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22(7), 861-876. https://doi.org/10.1002/acp.1391
  3. van Gog, T., & Sweller, J. (2015). Not new, but nearly forgotten: The testing effect decreases or even disappears as the complexity of learning materials increases. Educational Psychology Review, 27(2), 247-264. https://doi.org/10.1007/s10648-015-9310-x
  4. Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331(6018), 772-775. https://doi.org/10.1126/science.1199327
  5. Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of Educational Psychology, 106(3), 849-858. https://doi.org/10.1037/a0035934
  6. Roelle, J., Roelle, D., & Berthold, K. (2019). Test-based learning: Inconsistent effects between higher- and lower-level test questions. The Journal of Experimental Education, 87(2), 299-313. https://doi.org/10.1080/00220973.2018.1434756
  7. Roelle, J., & Nückles, M. (2019). Generative learning versus retrieval practice in learning from text: The cohesion and elaboration of the text matters. Journal of Educational Psychology, 111(8), 1341-1361. https://doi.org/10.1037/edu0000345
  8. Hiller, S., Rumann, S., Berthold, K., & Roelle, J. (2020). Example-based learning: Should learners receive closed-book or open-book self-explanation prompts? Instructional Science, 48(6), 623-649. https://doi.org/10.1007/s11251-020-09523-4
  9. Ebersbach, M. (2020). Access to the learning material enhances learning by means of generating questions: Comparing open- and closed-book conditions. Trends in Neuroscience and Education, 19, 100130. https://doi.org/10.1016/j.tine.2020.100130
  10. Waldeyer, J., Heitmann, S., Moning, J., & Roelle, J. (2020). Can generative learning tasks be optimized by incorporation of retrieval practice? Journal of Applied Research in Memory and Cognition, 9(3), 355-369. https://doi.org/10.1016/j.jarmac.2020.05.001
  11. Roelle, J., & Renkl, A. (2020). Does an option to review instructional explanations enhance example-based learning? It depends on learners' academic self-concept. Journal of Educational Psychology, 112(1), 131-147. https://doi.org/10.1037/edu0000365
  12. Arnold, K. M., Eliseev, E. D., Stone, A. R., McDaniel, M. A., & Marsh, E. J. (2021). Two routes to the same place: Learning from quick closed-book essays versus open-book essays. Journal of Cognitive Psychology, 33(3), 229-246. https://doi.org/10.1080/20445911.2021.1903011
  13. O'Day, G. M., & Karpicke, J. D. (2021). Comparing and combining retrieval practice and concept mapping. Journal of Educational Psychology, 113(5), 986-997. https://doi.org/10.1037/edu0000486
  14. Heitmann, S., Grund, A., Fries, S., Berthold, K., & Roelle, J. (2022). The quizzing effect depends on hope of success and can be optimized by cognitive load-based adaptation. Learning and Instruction, 77, 101526. https://doi.org/10.1016/j.learninstruc.2021.101526
  15. Sibley, L., Fiorella, L., & Lachner, A. (2022). It's better when I see it: Students benefit more from open-book than closed-book teaching. Applied Cognitive Psychology, 36(6), 1347-1355. https://doi.org/10.1002/acp.4017
  16. Wenzel, K., Schweppe, J., & Rummer, R. (2022). Are open-book tests still as effective as closed-book tests even after a delay of 2 weeks? Applied Cognitive Psychology, 36(3), 699-707. https://doi.org/10.1002/acp.3943
  17. Roelle, J., Froese, L., Krebs, R., Obergassel, N., & Waldeyer, J. (2022). Sequence matters! Retrieval practice before generative learning is more effective than the reverse order. Learning and Instruction, 80, 101634. https://doi.org/10.1016/j.learninstruc.2022.101634
  18. Roelle, J., Endres, T., Abel, R., Obergassel, N., Nückles, M., & Renkl, A. (2023). Happy together? On the relationship between research on retrieval practice and generative learning using the case of follow-up learning tasks. Educational Psychology Review, 35(4), Article 102. https://doi.org/10.1007/s10648-023-09810-9
  19. Buchin, Z. L., & Mulligan, N. W. (2023). Retrieval-based learning and prior knowledge. Journal of Educational Psychology. https://doi.org/10.1037/edu0000773
  20. Endres, T., Carpenter, S. K., & Renkl, A. (2024). Constructive retrieval: Benefits for learning, motivation, and metacognitive monitoring. Learning and Instruction, 94, 101974. https://doi.org/10.1016/j.learninstruc.2024.101974
  21. Macaluso, J. A., & Fraundorf, S. H. (2025). Do differences in topic knowledge matter? An experimental investigation into topic knowledge as a possible moderator of the testing effect. Memory, 33(5), 566-587. https://doi.org/10.1080/09658211.2025.2500538
  22. Kim, S., Abel, R., Leutner, D., Schmiemann, P., & Roelle, J. (2025). Can the effects of generative drawing on lasting learning be optimized through integration of retrieval practice? Learning and Instruction, 98, 102143. https://doi.org/10.1016/j.learninstruc.2025.102143
  23. Permzadian, V., & Cho, K. W. (2025). Does test format affect learning? A meta-analysis comparing the effect of closed-book and open-book examinations. Assessment & Evaluation in Higher Education, 50(7), 1098-1118. https://doi.org/10.1080/02602938.2025.2502590
  24. Rummer, R., Schweppe, J., & Schwede, A. (2019). Open-book versus closed-book tests in university classes: A field experiment. Frontiers in Psychology, 10, 463. https://doi.org/10.3389/fpsyg.2019.00463
  25. Agarwal, P. K., & Roediger, H. L., II. (2011). Expectancy of an open-book test decreases performance on a delayed closed-book test. Memory, 19(8), 836-852. https://doi.org/10.1080/09658211.2011.613840