ManyClasses 1: Assessing the Generalizable Effect of Immediate Feedback Versus Delayed Feedback Across Many College Classes

Advances in Methods and Practices in Psychological Science

Volumn 4, Issue 3, 2021, Pages

  Fyfe, Emily R ^a   de Leeuw, Joshua R ^b   Carvalho, Paulo F ^c   Goldstone, Robert L ^a   Sherman, Janelle ^a   Admiraal, David ^d   Alford, Laura K ^e   Bonner, Alison ^f   Brassil, Chad E ^g   Brooks, Christopher A ^e   Carbonetto, Tracey ^f   Chang, Sau Hou ^h   Cruz, Laura ⁱ   Czymoniewicz Klippel, Melina ⁱ   Daniel, Frances ^j   Driessen, Michelle ^k   Habashy, Noel ^l   Hanson Bradley, Carrie L ^d   Hirt, Edward R ^a   Carbonell, Virginia Hojas ^a   Jackson, Daniel K ^f   Jones, Shay ^m   Keagy, Jennifer L ^m   Keith, Brandi ⁿ   Malmquist, Sarah J ^k   McQuarrie, Barry ^o   Metzger, Kelsey J ^p   Min, Maung K ^f   Patil, Sameer ^q   Patrick, Ryan S ^d,r   Pelaprat, Etienne ^s   Petrunich Rutherford, Maureen L ^j   Porter, Meghan R ^a   Prescott, Kristina ^k   Reck, Cathrine ^a   Renner, Terri ^a   Robbins, Eric ^t   Smith, Adam R ^u   Stuczynski, Phil ^t   Thompson, Jaye ^k   Tsotakos, Nikolaos ^m   Turk, Judith K ^v   Unruh, Kyle ^s   Webb, Jennifer D ^w   Whitehead, Stephanie N ^x   Wisniewski, Elaine C ^y   Zhang, Ke Anne ^a   Motz, Benjamin A ^a   more..

^a INDIANA UNIVERSITY   (United States)

^b VASSAR COLLEGE   (United States)

^c Carnegie Mellon University ^*  (United States)

^d UNIVERSITY OF NEBRASKA LINCOLN   (United States)

^e UNIVERSITY OF MICHIGAN   (United States)

^f PENNSYLVANIA STATE UNIVERSITY   (United States)

^g UNIVERSITY OF NEBRASKA   (United States)

^h INDIANA UNIVERSITY SOUTHEAST   (United States)

ⁱ PENNSYLVANIA STATE UNIVERSITY   (United States)

^j INDIANA UNIVERSITY NORTHWEST   (United States)

^k UNIVERSITY OF MINNESOTA   (United States)

^l PENNSYLVANIA STATE UNIVERSITY   (United States)

^m PENNSYLVANIA STATE UNIVERSITY   (United States)

ⁿ INDIANA UNIVERSITY KOKOMO   (United States)

^o UNIVERSITY OF MINNESOTA   (United States)

^p UNIVERSITY OF MINNESOTA   (United States)

^q INDIANA UNIVERSITY   (United States)

^r UNIVERSITY OF NEBRASKA LINCOLN   (United States)

^s Unizin   (United States)

^t Penn State Erie   (United States)

^u INDIANA UNIVERSITY KOKOMO   (United States)

^v UNIVERSITY OF NEBRASKA LINCOLN   (United States)

^w UNIVERSITY OF MINNESOTA DULUTH   (United States)

^x INDIANA UNIVERSITY EAST   (United States)

^y The Department of Civil and Environmental Engineering   (United States)

more..

Author keywords

education; evidence based practices; experiment; feedback; reproducibility

Indexed keywords

EID: 85110363528     PISSN: 25152459     EISSN: 25152467     Source Type: Journal    
DOI: 10.1177/25152459211027575     Document Type: Article

References (45)

1. Anderson L. S., Healy A. F., Kole J. A., Bourne L. E., Jr. (2013). The clicker technique: Cultivating efficient teaching and successful learning. Applied Cognitive Psychology, 27, 222–234. https://doi.org/10.1002/acp.2899
2. Andrews T. C., Lemons P. P., (2015). It’s personal: Biology instructors prioritize personal evidence over empirical evidence in teaching decisions. CBE - Life Sciences Education, 14, 1–18. https://doi.org/10.1187/cbe.14-05-0084
3. Baribault B., Donkin C., Little D. R., Trueblood J. S., Oravecz Z., van Ravenzwaaij D., White C. N., De Boeck P., Vandekerckhove J., (2017). Metastudies for robust tests of theory. Proceedings of the National Academy of Sciences, USA, 11, 2607–2612. https://doi.org/10.1083/pnas.1708285114
4. Benassi V. A., Overson C. E., Hakala C. M., (2014). Applying science of learning in education: Infusing psychological science into the curriculum. American Psychological Association.
5. Bloom H. S., Spybrook J., (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 4, 877–902. https://doi.org/10.1080/19345747.2016.1271069
6. Bohn M., Schmitt V., Sanchez-Amaro A., Keupp S., Hopper L., Völter C., Altschul D., Fischer J., Fichtel C., Beran M. J., Kano F., Call J., Watzek J., Joly M., Hernandez-Aguilar R. A., (2019). Establishing an infrastructure for collaboration in primate cognition research. OSF. https://doi.org/10.31234/osf.io/3xu7q
7. Booth J. L., McGinn K. M., Barbieri C., Begolli K. N., Chang B., Miller-Coto D., Young L. K., Davenport J., (2017). Evidence for cognitive science principles that impact learning in mathematics. In Geary D. C., Berch D. B., Ochsendorf R., Koepke K. M., (Eds.), Acquisition of complex arithmetic skills and higher-order mathematics concepts (Vol. 3, pp. 297–325). Academic Press.
8. Booth J. L., Oyer M., Paré-Blagoev J., Elliot A. J., Barbieri C., Augustine A., Koedinger K., (2015). Learning algebra by example in real-world classrooms. Journal of Research on Educational Effectiveness, 8, 530–551. https://doi.org/10.1080/19345747.2015.1055636
9. 2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230
10. Butler A. C., Woodward N. R., (2018). Towards a consilience in the use of task-level feedback to promote learning. In Federmeier K. D., (Ed.), Psychology of learning and motivation (pp. 1–38). Academic Press. https://doi.org/10.1016/bs.plm.2018.09.001
11. Carvalho P. F., Braithwaite D. W., de Leeuw J. R., Motz B. A., Goldstone R. L., (2016). An in vivo study of self-regulated study sequencing in introductory psychology courses. PLOS ONE, 11, Article e0152115. https://doi.org/10.1371/journal.pone.0152115
12. Cronbach L. J., (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116–127.
13. Dabbagh N., Bass R., Bishop M., Costelloe S., Cummings K., Freeman B., Frye M., Picciano A. G., Porowski A., Sparrow J., Wilson S. J., (2019). Using technology to support postsecondary student learning: A practice guide for college and university administrators, advisors, and faculty. What Works Clearinghouse, National Center for Education Evaluation and Regional Assistance.
14. Denwood M. J., (2016). runjags: An R package providing interface utilities, model templates, parallel computing methods and additional distributions for MCMC models in JAGS. Journal of Statistical Software, 71(9), 1–25. https://doi.org/10.18637/jss.v071.i09
15. Frank M. C., Bergelson E., Bergmann C., Cristia A., Floccia C., Gervain J., Hamlin J. K., Hannon E. E., Kline M., Levelt C., Lew-Williams C., Nazzi T., Panneton R., Rabagliati H., Soderstrom M., Sullivan J., Waxman S., Yurovsky D., (2017). A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy, 22, 421–435. https://doi.org/10.1111/infa.12182
16. Freeman S., Eddy S. L., McDonough M., Smith M. K., Okoroafor N., Jordt H., Wenderoth M. P., (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, USA, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111
17. Fyfe E. R., Brown S. A., (2018). Feedback influences children’s reasoning about math equivalence: A meta-analytic review. Thinking and Reasoning, 24, 157–178. https://doi.org/10.1080/13546783.2017.1359208
18. Gelman A., Goodrich B., Garby J., Vehtari A., (2019). R-squared for Bayesian regression models. The American Statistician, 73(3), 307–309. https://doi.org/10.1080/00031305.2018.1549100
19. Gelman A., Rubin D. B., (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
20. Gurung R. A., Burns K., (2019). Putting evidence-based claims to the test: A multi-site classroom study of retrieval practice and spaced practice. Applied Cognitive Psychology, 33(5), 732–743. https://doi.org/10.1002/acp.3507
21. Hattie J., Timperley H., (2007). The power of feedback. Review of Educational Research, 77, 81–112. https://doi.org/10.3102/003465430298487
22. Hulleman C. S., Cordray D. S., (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88–110. https://doi.org/10.1080/19345740802539325
23. Klein R. A., Ratliff K. A., Vianello M., Adams R. B., Jr.Bahník Š., Bernstein M. J., Bocian K., Brandt M. J., Brooks B., Brumbaugh C. C., Cemalcilar Z., Chandler J., Cheong W., Davis W. E., Devos T., Eisner M., Frankowska N., Furrow D., Galliani E. M... Nosek B. A., (2014). Investigating variation in replicability. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178
24. Klein R. A., Vianello M., Hasselman F., Adams B. G., Adams R. B., Jr.Alper S., Aveyard M., Axt J. R., Babalola M. T., Bahník Š., Batra R., Berkics M., Bernstein M. J., Berry D. R., Bialobrzeska O., Binan E. D., Bocian K., Brandt M. J., Busching R... Nosek B. A., (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490. https://doi.org/10.1177/2515245918810225
25. Kluger A. N., DeNisi A., (1996). Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254–284. https://doi.org/10.1037/0033-2909.119.2.254
26. Knight J. K., Wood W. B., (2005). Teaching more by lecturing less. Cell Biology Education, 4, 298–310.
27. Koedinger K. R., Booth J. L., Klahr D., (2013). Instructional complexity and the science to constrain it. Science, 342(6161), 935–937.
28. Kruschke J. K., (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan. Academic Press.
29. Kruschke J. K., Liddell T. M., (2018a). Bayesian data analysis for newcomers. Psychonomic Bulletin & Review, 25(1), 155–177. https://doi.org/10.3758/s13423-017-1272-1
30. Kruschke J. K., Liddell T. M., (2018b). The Bayesian new statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. https://doi.org/10.3758/s13423-016-1221-4
31. Kulik J. A., Kulik C. C., (1988). Timing of feedback and verbal learning. Review of Educational Research, 58, 79–97. https://doi.org/10.3102/00346543058001079
32. LeBel E. P., Berger D., Campbell L., Loving T. J., (2017). Falsifiability is not optional. Journal of Personality and Social Psychology, 113, 254–261. https://doi.org/10.1037/pspi0000106
33. Makel M. C., Smith K. N., McBee M. T., Peters S. J., Miller E. M., (2019). A path to greater credibility: Large-scale collaborative education research. AERA Open, 5(4), 1–15. https://doi.org/10.1177/2332858419891963
34. Motz B. A., Carvalho P. F., de Leeuw J. R., Goldstone R. L., (2018). Embedding experiments: Staking causal inference in authentic educational contexts. Journal of Learning Analytics, 5, 47–59. https://doi.org/10.18608/jla.2018.52.4
35. Mullaney K. M., Carpenter S. K., Grotenhuis C., Burianek S., (2014). Waiting for feedback helps if you want to know the answer: The role of curiosity in the delay-of-feedback benefit. Memory & Cognition, 42(8), 1273–1284. https://doi.org/10.3758/s13421-014-0441-y
36. Mullet H. G., Butler A. C., Verdin B., von Borries R., Marsh E. J., (2014). Delaying feedback promotes transfer of knowledge despite student preferences to receive feedback immediately. Journal of Applied Research in Memory and Cognition, 3, 222–229. https://doi.org/10.1016/j.jarmac.2014.05.001
37. National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. The National Academies Press. https://doi.org/10.17226/24783
38. Plummer M., (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Hornik K., Leisch F., Zeileis A., (Eds.), Proceedings of the 3rd International Workshop of Distributed Statistical Computing (DSC 2003) (Vol. 124, pp. 1–10). Vienna, Austria: Technische Universität Wien.
39. Rohrer D., Dedrick R. F., Hartwig M. K., Cheung C.-N., (2020). A randomized controlled trial of interleaved mathematics practice. Journal of Educational Psychology, 112(1), 40–52. https://doi.org/10.1037/edu0000367
40. 2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8(1), Article 79. https://doi.org/10.1186/1471-2288-8-79
41. Skinner B. F., (1954). The science of learning and the art of teaching. Harvard Educational Review, 24, 86–97. https://doi.org/1955-02985-001
42. Tipton E., Hedges L. V., (2017). The role of the sample in estimating and explaining treatment effect heterogeneity. Journal of Research on Educational Effectiveness, 10(4), 903–906. https://doi.org/10.1080/19345747.2017.1364563
43. Wakeling V., Robertson P. R., (2017). A comparison of student behavior and performance between an instructor-regulated versus student-regulated online undergraduate finance course. American Journal of Educational Research, 5, 863–870. https://doi.org/10.12691/education-5-8-5
44. Wasserstein R. L., Schirm A. L., Lazar N. A., (2019). Moving to a world beyond “p < 0.05.” The American Statistician, 73(Suppl. 1), 1–19. https://doi.org/10.1080/00031305.2019.1583913
45. Yarkoni T., (2020). The generalizability crisis. Behavioral and Brain Sciences. Advance online publication. https://doi.org/10.1017/S0140525X20001685