Not New, but Nearly Forgotten: the Testing Effect Decreases or even Disappears as the Complexity of Learning Materials Increases

Educational Psychology Review

Volumn 27, Issue 2, 2015, Pages 247-264

  van Gog, Tamara ^a,b   Sweller, John ^c  

^a UTRECHT UNIVERSITY   (Netherlands)

^b ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Complex learning tasks; Problem solving; Retrieval practice; Testing effect; Worked examples

Indexed keywords

EID: 84929841158     PISSN: 1040726X     EISSN: 1573336X     Source Type: Journal    
DOI: 10.1007/s10648-015-9310-x     Document Type: Review

References (87)

1. Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L., & McDermott, K. B. (2008). Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22, 861–876. doi:10.1002/acp.1391.
2. Agarwal, P. K., Bain, P. M., & Chamberlain, R. W. (2012). The value of applied research: retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review, 24, 437–448. doi:10.1007/s10648-012-9210-2.
3. Anderson, J. R. (1982). Acquisition of a cognitive skill. Psychological Review, 89, 369–406. doi:10.1037/0033-295X.89.4.369.
4. Arnold, K. M., & McDermott, K. B. (2013a). Free recall enhances subsequent learning. Psychonomic Bulletin & Review, 20, 507–513. doi:10.3758/s13423-012-0370-3.
5. Arnold, K. M., & McDermott, K. B. (2013b). Test-potentiated learning: distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 940–945. doi:10.1037/a0029199.
6. Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of Educational Psychology, 106, 849–858. doi:10.1037/a0035934.
7. Bouwmeester, S., & Verkoeijen, P. P. J. L. (2011). Why do some children benefit more from testing than others? Gist trace processing to explain the testing effect. Journal of Memory and Language, 65, 32–41. doi:10.1016/j.jml.2011.02.005.
8. Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(5), 1118–1133. doi:10.1037/a0019902.
9. Butler, A. C., & Roediger, H. L. (2007). Testing improves long-term retention in a simulated classroom setting. European Journal of Cognitive Psychology, 19, 514–527. doi:10.1080/09541440701326097.
10. Carpenter, S. K. (2009). Cue strength as a moderator of the testing effect: the benefits of elaborative retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 1563–1569. doi:10.1037/a0017021.
11. Carpenter, S. K. (2012). Testing enhances the transfer of learning. Current Directions in Psychological Science, 21, 279–283. doi:10.1177/0963721412452728.
12. Carpenter, S. K., & DeLosh, E. L. (2006). Impoverished cue support enhances subsequent retention: support for the elaborative retrieval explanation of the testing effect. Memory & Cognition, 34, 268–276. doi:10.3758/BF03193405.
13. Carpenter, S. K., & Kelly, J. W. (2012). Tests enhance retention and transfer of spatial learning. Psychonomic Bulletin & Review, 19, 443–448. doi:10.3758/s13423-012-0221-2.
14. Carpenter, S. K., Pashler, H., & Vul, E. (2006). What types of learning are enhanced by a cued recall test? Psychonomic Bulletin & Review, 13, 826–830. doi:10.3758/BF03194004.
15. Carpenter, S. K., Pashler, H., Wixted, J. T., & Vul, E. (2008). The effects of tests on learning and forgetting. Memory & Cognition, 36, 438–448. doi:10.3758/MC.36.2.438.
16. Carpenter, S. K., Pashler, H., & Cepeda, N. J. (2009). Using tests to enhance 8th grade students’ retention of U.S. history facts. Applied Cognitive Psychology, 23, 760–771. doi:10.1002/acp.1507.
17. Carpenter, S. K., Sachs, R. E., Martin, B., Schmidt, K., & Looft, R. (2012). Learning new vocabulary in German: the effects of inferring word meanings, type of feedback, and time of test. Psychonomic Bulletin & Review, 19, 81–86. doi:10.3758/s13423-011-0185-7.
18. Coane, J. H. (2013). Retrieval practice and elaborative encoding benefit memory in younger and older adults. Journal of Applied Research in Memory and Cognition, 2, 95–100. doi:10.1016/j.jarmac.2013.04.001.
19. Coppens, L. C., Verkoeijen, P. P. J. L., & Rikers, R. M. J. P. (2011). Learning Adinkra symbols: the effect of testing. Journal of Cognitive Psychology, 23, 351–357. doi:10.1080/20445911.2011.507188.
20. Cranney, J., Ahn, M., McKinnon, R., Morris, S., & Watts, K. (2009). The testing effect, collaborative learning, and retrieval-induced facilitation in a classroom setting. European Journal of Cognitive Psychology, 21, 919–940. doi:10.1080/09541440802413505.
21. De Jonge, M. O., Tabbers, H. K., & Rikers, R. M. J. P. (2015). The effect of testing on the retention of coherent and incoherent text material. Educational Psychology Review. doi:10.1007/s10648-015-9300-z.
22. Efklides, A. (2006). Metacognition and affect: what can metacognitive experiences tell us about the learning process? Educational Research Review, 1, 3–14. doi:10.1016/j.edurev.2005.11.001.
23. Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity: eight learning strategies that promote understanding. New York: Cambridge University Press.
24. Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, 6(40).
25. Glover, J. A. (1989). The “testing” phenomenon: not gone but nearly forgotten. Journal of Educational Psychology, 81, 392–399. doi:10.1037/0022-0663.81.3.392.
26. Goossens, N. A. M. C., Camp, G., Verkoeijen, P. P. J. L., & Tabbers, H. K. (2014). The effect of retrieval practice in primary school vocabulary learning. Applied Cognitive Psychology, 28, 135–142. doi:10.1002/acp.2956.
27. Grimaldi, P. J., & Karpicke, J. D. (2014). Guided retrieval practice of educational materials using automated scoring. Journal of Educational Psychology, 106, 58–68. doi:10.1037/a0033208.
28. Jang, Y., Wixted, J. T., Pecher, D., Zeelenberg, R., & Huber, D. E. (2012). Decomposing the interaction between retention interval and study/test practice: the role of retrievability. The Quarterly Journal of Experimental Psychology, 65, 962–975. doi:10.1080/17470218.2011.638079.
29. Johnson, C. I., & Mayer, R. E. (2009). A testing effect with multimedia learning. Journal of Educational Psychology, 101, 621–629. doi:10.1037/a0015183.
30. Kang, S. H. K. (2010). Enhancing visuo-spatial learning: the benefit of retrieval practice. Memory & Cognition, 38, 1009–1017. doi:10.3758/MC.38.8.1009.
31. Kang, S. H. K., & Pashler, H. (2014). Is the benefit of retrieval practice modulated by motivation? Journal of Applied Research in Memory and Cognition, 3, 183–188. doi:10.1016/j.jarmac.2014.05.006.
32. Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modulate the effect of testing on memory retention. European Journal of Cognitive Psychology, 19, 528–558. doi:10.1080/09541440601056620.
33. Kang, S. H. K., McDaniel, M. A., & Pashler, H. (2011). Effects of testing on learning of functions. Psychonomic Bulletin and Review, 18, 998–1005. doi:10.3758/s13423-011-0113-x.
34. Karpicke, J. D., & Grimaldi, P. J. (2012). Retrieval-based learning: a perspective for enhancing meaningful learning. Educational Psychology Review, 24, 401–418. doi:10.1007/s10648-012-9202-2.
35. Karpicke, J. D., & Roediger, H. L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57, 151–162. doi:10.1016/j.jml.2006.09.004.
36. Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319, 966–968. doi:10.1126/science.1152408.
37. Karpicke, J. D., Butler, A. C., & Roediger, H. L. (2009). Metacognitive strategies in student learning: do students practice retrieval when they study on their own? Memory, 17, 471–479. doi:10.1080/09658210802647009.
38. Karpicke, J. D., Blunt, J. R., Smith, M. A., & Karpicke, S. S. (2014a). Retrieval-based learning: the need for guided retrieval in elementary school children. Journal of Applied Research in Memory and Cognition, 3, 198–206. doi:10.1016/j.jarmac.2014.07.008.
39. Karpicke, J. D., Lehman, M., & Aue, W. R. (2014b). Retrieval-based learning: an episodic context account. In B. H. Ross (Ed.), Psychology of learning and motivation (Vol. 61, pp. 237–284). San Diego: Elsevier Academic. doi:10.1016/B978-0-12-800283-4.00007-1.
40. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19, 239–264. doi:10.1007/s10648-007-9049-0.
41. Kornell, N., & Son, L. K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17, 493–501. doi:10.1080/09658210902832915.
42. Kornell, N., Hays, M. J., & Bjork, R. A. (2009). Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 989–998. doi:10.1037/a0015729.
43. Kornell, N., Klein, P. J., & Rawson, K. A. (2015). Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 283–294. doi:10.1037/a0037850.
44. Kühn, A. (1914). Über Einprägung durch Lesen und durch Rezitieren [On imprinting through reading and reciting]. Zeitschrift für Psychologie, 68, 396–481. Available from: http://babel.hathitrust.org/cgi/pt?id=njp.32101074935824;view=1up;seq=422.
45. Larsen, D. P., Butler, A. C., & Roediger, H. L. (2009). Repeated testing improves long-term retention relative to repeated study: a randomized controlled trial. Medical Education, 43, 1174–1181. doi:10.1111/j.1365-2923.2009.03518.x.
46. Leahy, W., Hanham, J., & Sweller, J. (2015). High element interactivity information during problem solving may lead to failure to obtain the testing effect. Educational Psychology Review. doi:10.1007/s10648-015-9296-4.
47. Leppink, J., Paas, F., van Gog, T., van der Vleuten, C., & van Merriënboer, J. J. G. (2014). Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction, 30, 32–42. doi:10.1016/j.learninstruc.2013.12.001.
48. Masson, M. E., & McDaniel, M. A. (1981). The role of organizational processes in long-term retention. Journal of Experimental Psychology: Human Learning and Memory, 7, 100–110. doi:10.1037/0278-7393.7.2.100.
49. McDaniel, M. A., & Einstein, G. O. (1989). Material-appropriate processing: a contextualist approach to reading and studying strategies. Educational Psychology Review, 1, 112–145. doi:10.1007/BF01326639.
50. McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the testing effect in the classroom. European Journal of Cognitive Psychology, 19, 494–513. doi:10.1080/09541440701326154.
51. McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: the effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399–414. doi:10.1037/a0021782.
52. McDaniel, M. A., Wildman, K. M., & Anderson, J. L. (2012). Using quizzes to enhance summative-assessment performance in a web-based class: an experimental study. Journal of Applied Research in Memory and Cognition, 1, 18–26. doi:10.1016/j.jarmac.2011.10.001.
53. McDermott, K. B., & Naaz, F. (2014). Is recitation an effective tool for adult learners? Journal of Applied Research in Memory and Cognition, 3, 207–213. doi:10.1016/j.jarmac.2014.06.006.
54. McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20, 3–21. doi:10.1037/xap0000004.
55. Metcalfe, J., & Kornell, N. (2003). The dynamics of learning and allocation of study time to a region of proximal learning. Journal of Experimental Psychology: General, 132, 530–542. doi:10.1037/0096-3445.132.4.530.
56. Pashler, H., Cepeda, N. J., Wixted, J. T., & Rohrer, D. (2005). When does feedback facilitate learning of words? Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 3–8. doi:10.1037/0278-7393.31.1.3.
57. Putnam, A. L., & Roediger, H. L. (2013). Does response mode affect amount recalled or the magnitude of the testing effect? Memory & Cognition, 41, 36–48. doi:10.3758/s13421-012-0245-x.
58. Pyc, M. A., & Rawson, K. A. (2009). Testing the retrieval effort hypothesis: does greater difficulty correctly recalling information lead to higher levels of memory? Journal of Memory and Language, 60, 437–447. doi:10.1016/j.jml.2009.01.004.
59. Pyc, M. A., & Rawson, K. A. (2010). Why testing improves memory: mediator effectiveness hypothesis. Science, 330, 335. doi:10.1126/science.1191465.
60. Pyc, M. A., & Rawson, K. A. (2012). Why is retrieval practice beneficial for memory? An evaluation of the mediator shift hypothesis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38, 737–746. doi:10.1037/a0026166.
61. Richland, L. E., Kornell, N., & Kao, L. S. (2009). The pretesting effect: do unsuccessful retrieval attempts enhance learning? Journal of Experimental Psychology: Applied, 15, 243–257. doi:10.1037/a0016496.
62. Roediger, H. L., & Karpicke, J. D. (2006a). Test-enhanced learning: taking memory tests improves long-term retention. Psychological Science, 17, 249–255. doi:10.1111/j.1467-9280.2006.01693.x.
63. Roediger, H. L., & Karpicke, J. D. (2006b). The power of testing memory: basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181–210. doi:10.1111/j.1745-6916.2006.00012.x.
64. Roediger, H. L., & Nestojko, J. F. (2015). The relative benefits of studying and testing on long-term retention. In J. G. W. Raaijmakers, A. H. Criss, R. L. Goldstone, R. M. Nosofsky, & M. Styvers (Eds.), Cognitive modeling in perception and memory: a festschrift for Richard M. Shiffrin (pp. 99–111). New York: Psychology.
65. Roediger, H. L., & Pyc, M. A. (2012a). Inexpensive techniques to improve education: applying cognitive psychology to enhance educational practice. Journal of Applied Research in Memory and Cognition, 1, 242–248. doi:10.1016/j.jarmac.2012.10.006.
66. Roediger, H. L., & Pyc, M. A. (2012b). Applying cognitive psychology to education: complexities and prospects. Journal of Applied Research in Memory and Cognition, 1, 263–265. doi:10.1016/j.jarmac.2012.10.006.
67. Roediger, H. L., Agarwal, P. K., Kang, S. H. K., & Marsh, E. J. (2010). Benefits of testing memory: best practices and boundary conditions. In G. M. Davies & D. B. Wright (Eds.), New frontiers in applied memory (pp. 13–49). Brighton: Psychology.
68. Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011a). Test-enhanced learning in the classroom: long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382–395. doi:10.1037/a0026252.
69. Roediger, H. L., Putnam, A. L., & Smith, M. A. (2011b). Ten benefits of testing and their applications to educational practice. In J. Mestre & B. Ross (Eds.), Psychology of learning and motivation: cognition in education (pp. 1–36). Oxford: Elsevier.
70. Rowland, C. A. (2014). The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychological Bulletin, 140, 1432–1463. doi:10.1037/a0037559.
71. Smith, M. A., & Karpicke, J. D. (2014). Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory, 22, 784–802. doi:10.1080/09658211.2013.831454.
72. Smith, M. A., Roediger, H. L., & Karpicke, J. D. (2013). Covert retrieval practice benefits retention as much as overt retrieval practice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1712–1725. doi:10.1037/a0033569.
73. Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641–656. doi:10.1037/h0063404.
74. Sweller, J. (2010). Element interactivity and intrinsic, extraneous and germane cognitive load. Educational Psychology Review, 22, 123–138. doi:10.1007/s10648-010-9128-5.
75. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.
76. Toppino, T. C., & Cohen, M. S. (2009). The testing effect and the retention interval: questions and answers. Experimental Psychology, 56, 252–257. doi:10.1027/1618-3169.56.4.252.
77. Tran, R., Rohrer, D., & Pashler, H. (2015). Retrieval practice: the lack of transfer to deductive inferences. Psychonomic Bulletin and Review, 22, 135–140. doi:10.3758/s13423-014-0646-x.
78. Tse, C. S., Balota, D. A., & Roediger, H. L. (2010). The benefits and costs of repeated testing: on the learning of face-name pairs in older adults. Psychology and Aging, 25, 833–845. doi:10.1037/a0019933.
79. Tulving, E. (1962). Subjective organization and effects of repetition in multi-trial free-recall learning. Journal of Verbal Learning and Verbal Behavior, 5, 193–197. doi:10.1016/S0022-5371(66)80016-6.
80. van Gog, T., & Kester, L. (2012). A test of the testing effect: acquiring problem-solving skills from worked examples. Cognitive Science, 36, 1532–1541. doi:10.1111/cogs.12002.
81. van Gog, T., Kester, L., & Paas, F. (2011). Effects of worked examples, example-problem, and problem-example pairs on novices’ learning. Contemporary Educational Psychology, 36, 212–218. doi:10.1016/j.cedpsych.2010.10.004.
82. van Gog, T., Kester, L., Dirkx, K., Hoogerheide, V., Boerboom, J., & Verkoeijen, P. P. J. L. (2015). Testing after worked example study does not enhance delayed problem-solving performance compared to restudy. Educational Psychology Review. doi:10.1007/s10648-015-9297-3
83. Verkoeijen, P. P. J. L., Bouwmeester, S., & Camp, G. (2012). A short term testing effect in cross-language recognition. Psychological Science, 23(6), 567–571. doi:10.1177/0956797611435132.
84. Weinstein, Y., McDermott, K. B., & Roediger, H. L. (2010). A comparison of study strategies for passages: re-reading, answering questions, and generating questions. Journal of Experimental Psychology: Applied, 16, 308–316. doi:10.1037/a0020992.
85. Wheeler, M. A., Ewers, M., & Buonanno, J. (2003). Different rates of forgetting following study versus test trials. Memory, 11, 571–580. doi:10.1080/09658210244000414.
86. Zaromb, F. M. (2010). Organizational processes contribute to the testing effect in free recall. Doctoral dissertation, Washington University in St. Louis.
87. Zaromb, F. M., & Roediger, H. L. (2010). The testing effect in free recall is associated with enhanced organizational processes. Memory & Cognition, 38, 995–1008. doi:10.3758/MC.38.8.995.