In defence of psychometric measurement: a systematic review of contemporary self-report feedback inventories

Educational Psychologist

Volumn 58, Issue 3, 2023, Pages 178-192

  Brown, Gavin T L ^a,b   Zhao, Anran ^c  

^a UNIVERSITY OF AUCKLAND   (New Zealand)

^b UMEÅ UNIVERSITY   (Sweden)

^c Ministry of Education   (New Zealand)

Author keywords

Feedback; measurement inventories; perceptions or conceptions or perspectives or orientation; psychometrics; self regulated learning and self regulation

Indexed keywords

EID: 85159637682     PISSN: 00461520     EISSN: None     Source Type: Journal    
DOI: 10.1080/00461520.2023.2208670     Document Type: Article

References (98)

1. * Adams, A.-M., Wilson, H., Money, J., Palmer-Conn, S., & Fearn, J. (2020). Student engagement with feedback and attainment: the role of academic self-efficacy. Assessment & Evaluation in Higher Education, 45 (2), 317–329. https://doi.org/10.1080/02602938.2019.1640184
2. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50 (2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
3. Ajzen, I. (2005). Attitudes, personality and behavior. (2nd ed.). Open University Press.
4. Allport, G. W. (1935). Attitudes. In C. Murchison (Ed.), A Handbook of Social Psychology (pp. 798–844). Clark University Press.
5. Ashford, S. J., Stobbeleir, K. D., & Nujella, M. (2016). To seek or not to seek: Is that the only question? Recent developments in feedback-seeking literature. Annual Review of Organizational Psychology and Organizational Behavior, 3 (1), 213–239. https://doi.org/10.1146/annurev-orgpsych-041015-062314
6. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice-Hall.
7. Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54 (2), 199–231. https://doi.org/10.1111/j.1464-0597.2005.00205.x
8. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
9. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111 (4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
10. Brown, G. T. L., Harris, L. R., O'Quin, C., & Lane, K. E. (2017). Using multi-group confirmatory factor analysis to evaluate cross-cultural research: Identifying and understanding non-invariance. International Journal of Research & Method in Education, 40 (1), 66–90. https://doi.org/10.1080/1743727X.2015.1070823
11. Brown, G. T. L., Peterson, E. R., & Irving, S. E. (2009). Beliefs that make a difference: Adaptive and maladaptive self-regulation in students’ conceptions of assessment. In D. M. McInerney, G. T. L. Brown, & G. A. D. Liem (Eds.), Student perspectives on assessment: What students can tell us about assessment for learning (pp. 159–186). Information Age Publishing.
12. * Brown, G. T. L., Peterson, E. R., & Yao, E. S. (2016). Student conceptions of feedback: Impact on self-regulation, self-efficacy, and academic achievement. The British Journal of Educational Psychology, 86 (4), 606–629. https://doi.org/10.1111/bjep.12126
13. Brown, G. T. L., & Shulruf, B. (2023). Response option design in surveys. In L. R. Ford & T. A. Scandura (Eds.), The SAGE Handbook of Survey Development and Application. Sage.
14. Bryant, F. B., & Yarnold, P. R. (1995). Principal-components analysis and exploratory and confirmatory factor analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 99–136). APA.
15. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65 (3), 245–281. https://doi.org/10.3102/00346543065003245
16. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures–the issue of partial measurement invariance. Psychological Bulletin, 105 (3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456
17. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multi-trait multi-method matrix. Psychological Bulletin, 56 (2), 81–105. https://doi.org/10.1037/h0046016
18. Carpenter, S. K., Witherby, A. E., & Tauber, S. K. (2020). On students’ (Mis)judgments of learning and teaching effectiveness. Journal of Applied Research in Memory and Cognition, 9 (2), 137–151. https://doi.org/10.1016/j.jarmac.2019.12.009
19. Cizek, G. J. (2020). Validity: An integrated approach to test score meaning and use. Routledge.
20. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Routledge.
21. Crafts, L. W., & Gilbert, R. W. (1935). The effect of knowledge of results on maze learning and retention. Journal of Educational Psychology, 26 (3), 177–187. https://doi.org/10.1037/h0061908
22. Cronbach, L. J. (1988). Five perspectives on the validity argument. In H. Wainer & H. Braun (Eds.), Test validity (pp. 3–17). Lawrence Erlbaum.
23. Cudeck, R., & MacCallum, R. C. (Eds.) (2007). Factor analysis at 100: Historical developments and future directions. LEA.
24. De Stobbeleir, K. E. M., Ashford, S. J., & Buyens, D. (2011). Self-regulation of creativity at work: The role of feedback-seeking behavior in creative performance. Academy of Management Journal, 54 (4), 811–831. https://doi.org/10.5465/amj.2011.64870144
25. DiStefano, C., Zhu, M., & Mîndrilă, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research & Evaluation, 14 (20), 1–11. https://doi.org/10.7275/da8t-4g52
26. Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5 (3), 69–106. https://doi.org/10.1111/j.1529-1006.2004.00018.x
27. Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. LEA.
28. Esarey, J., & Valdes, N. (2020). Unbiased, reliable, and valid student evaluations can still be unfair. Assessment & Evaluation in Higher Education, 45 (8), 1106–1120. https://doi.org/10.1080/02602938.2020.1724875
29. Fan, X., Miller, B. C., Park, K.-E., Winward, B. W., Christensen, M., Grotevant, H. D., & Tai, R. H. (2006). An exploratory study about inaccuracy and invalidity in adolescent self-report surveys. Field Methods, 18 (3), 223–244. https://doi.org/10.1177/152822X06289161
30. Fatima, S., Ali, M., & Saad, M. I. (2022). The effect of students’ conceptions of feedback on academic self-efficacy and self-regulation: evidence from higher education in Pakistan. Journal of Applied Research in Higher Education, 14 (1), 180–199. https://doi.org/10.1108/JARHE-07-2020-0209
31. Fong, C. J., & Schallert, D. L. (2023). “Feedback to the future”: Advancing motivational and emotional perspectives in feedback research. Educational Psychologist, 1–16. https://doi.org/10.1080/00461520.2022.2134135
32. Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3 (4), 456–465. https://doi.org/10.1177/2515245920952393
33. Gibbs, G., & Simpson, C. (2003). Measuring the response of students to assessment: The assessment experience questionnaire. Paper Presented at the 11th Improving Student Learning Symposium.
34. Guo, W. (2020). Grade-level differences in teacher feedback and students’ self-regulated learning. Frontiers in Psychology, 11, 783. https://doi.org/10.3389/fpsyg.2020.00783
35. Harris, L. R., & Brown, G. T. L. (2010). Mixing interview and questionnaire methods: Practical problems in aligning data. Practical Assessment, Research and Evaluation, 15 (1), 2. https://doi.org/10.7275/959j-ky83
36. Harris, L. R., & Brown, G. T. L. (2018). Understanding and supporting student self-assessment. Routledge.
37. Harris, L. R., Brown, G. T. L., & Dargusch, J. (2018). Not playing the game: student assessment resistance as a form of agency. The Australian Educational Researcher, 45 (1), 125–140. https://doi.org/10.1007/s13384-018-0264-0
38. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77 (1), 81–112. https://doi.org/10.3102/003465430298487
39. Hattie, J., Hodis, F. A., & Kang, S. H. K. (2020). Theories of motivation: Integration and ways forward. Contemporary Educational Psychology, 61, 101865. https://doi.org/10.1016/j.cedpsych.2020.101865
40. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? The Behavioral and Brain Sciences, 33 (2–3), 61–135. https://doi.org/10.1017/S0140525X0999152X
41. Hernández Rivero, V. M., Bonilla, P. J. S., & Alonso, J. J. S. (2021). Feedback and self-regulated learning in higher education. Revista de Investigación Educativa, 39 (1), 227–248. https://doi.org/10.6018/rie.423341
42. * Howell, J. L., & Shepperd, J. A. (2016). Establishing an information avoidance scale. Psychological Assessment, 28 (12), 1695–1708. https://doi.org/10.1037/pas0000315
43. International Test Commission. (2017). The ITC guidelines for translating and adapting tests. https://tinyurl.com/t6z2v42n
44. Irving, S. E., Peterson, E. R., & Brown, G. T. L. (2007). Student Conceptions of Feedback: A study of New Zealand secondary students. Paper Presented to the Biennial Conference of the European Association for Research in Learning and Instruction, Budapest, Hungary.
45. * Jellicoe, M., & Forsythe, A. (2019). The development and validation of the feedback in learning scale (FLS). Frontiers in Education, 4, 84. https://doi.org/10.3389/feduc.2019.00084
46. Jones, L. V., & Thissen, D. (2006). A history and overview of psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 1–27). Elsevier.
47. Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70 (351a), 631–639. https://doi.org/10.1080/01621459.1975.10482485
48. Jöreskog, K. G., & van Thillo, M. (1972). LISREL: A general computer program for estimating a linear structural equation system involving multiple indicators of unmeasured variables (RB-72-56). Educational Testing Service.
49. Joughin, G., Boud, D., Dawson, P., & Tai, J. (2021). What can higher education learn from feedback seeking behaviour in organisations? Implications for feedback literacy. Assessment & Evaluation in Higher Education, 46 (1), 80–91. https://doi.org/10.1080/02602938.2020.1733491
50. Karabenick, S. A., & Berger, J.-L. (2013). Help seeking as a self-regulated learning strategy. In Applications of self-regulated learning across diverse disciplines: A tribute to Barry J. Zimmerman (pp. 237–261). Information Age Publishing.
51. Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., Groot, E. D., Gilbert, M. C., Musu, L., Kempler, T. M., & Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42 (3), 139–151. https://doi.org/10.1080/00461520701416231
52. Kim, S.-I., Lee, M.-J., Chung, Y., & Bong, M. (2010). Comparison of brain activation during norm-referenced versus criterion-referenced feedback: The role of perceived competence and performance-approach goals. Contemporary Educational Psychology, 35 (2), 141–152. https://doi.org/10.1016/j.cedpsych.2010.04.002
53. * King, P. E., Schrodt, P., & Weisel, J. J. (2009). The instructional feedback orientation scale: Conceptualizing and validating a new measure for assessing perceptions of instructional feedback. Communication Education, 58 (2), 235–261. https://doi.org/10.1080/03634520802515705
54. Kirschner, P. A., & Hendrick, C. (2020). Why independent learning is not a good way to become an independent learner. In P. A. Kirschner & C. Hendrick (Eds.), How learning happens: Seminal works in educational psychology and what they mean in practice (pp. 66–73). Routledge.
55. * König, N., & Puca, R. M. (2019). The German feedback socialization inventory. European Journal of Psychological Assessment, 35 (4), 544–554. https://doi.org/10.1027/1015-5759/a000409
56. Krosnick, J. A., & Presser, S. (2009). Question and questionnaire design. Elsevier.
57. * Kyaruzi, F., Strijbos, J. W., Ufer, S., & Brown, G. T. L. (2019). Students’ formative assessment perceptions, feedback use and mathematics performance in secondary schools in Tanzania. Assessment in Education: Principles, Policy and Practice, 26 (3), 278–302. https://doi.org/10.1080/0969594X.2019.1593103
58. Leckey, J., & Neill, N. (2001). Quantifying quality: The importance of student feedback. Quality in Higher Education, 7 (1), 19–32. https://doi.org/10.1080/13538320120045058
59. * Linderbaum, B. A., & Levy, P. E. (2010). The development and validation of the feedback orientation scale (FOS). Journal of Management, 36 (6), 1372–1405. https://doi.org/10.1177/0149206310373145
60. Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26 (12), 1827–1832. https://doi.org/10.1177/0956797615616374
61. Lipnevich, A. A., & Panadero, E. (2021). A review of feedback models and theories: Descriptions, definitions, and conclusions. Frontiers in Education, 6, 100416. https://doi.org/10.3389/feduc.2021.720195
62. Lipnevich, A. A., Berg, D. A. G., & Smith, J. K. (2016). Toward a model of student response to feedback. In G. T. L. Brown & L. R. Harris (Eds.), The handbook of human and social conditions in assessment (pp. 169–185). Routledge.
63. Little, T., Dawson, P., Boud, D., & Tai, J. (2023). Can students’ feedback literacy be improved? A scoping review of interventions. Assessment & Evaluation in Higher Education, 1–14. https://doi.org/10.1080/02602938.2023.2177613
64. Lundgren, E., & Eklöf, H. (2020). Within-item response processes as indicators of test-taking effort and motivation. Educational Research and Evaluation, 26 (5–6), 275–301. https://doi.org/10.1080/13803611.2021.1963940
65. Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research. Perspectives on Psychological Science, 7 (6), 537–542. https://doi.org/10.1177/1745691612460688
66. Marsh, H. W., Hau, K.-T., Artelt, C., Baumert, J., & Peschar, J. L. (2006). OECD's brief self-report measure of educational psychology’s most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6 (4), 311–360. https://doi.org/10.1207/s15327574ijt0604_1
67. McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Instrument development in the affective domain: school and corporate applications. Springer.
68. McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23 (3), 412–433. https://doi.org/10.1037/met0000144
69. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). MacMillan.
70. Meyer, G. M., Marco-Pallarés, J., Boulinguez, P., & Sescousse, G. (2021). Electrophysiological underpinnings of reward processing: Are we exploiting the full potential of EEG? NeuroImage, 242, 118478. https://doi.org/10.1016/j.neuroimage.2021.118478
71. Murphy, P. K., & Alexander, P. A. (2000). A motivated exploration of motivation terminology. Contemporary Educational Psychology, 25 (1), 3–53. https://doi.org/10.1006/ceps.1999.1019
72. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31 (2), 199–218. https://doi.org/10.1080/03075070600572090
73. Neisser, U., Boodoo, G., Bouchard, T. J., Jr., Boykin, A. W., Brody, N., Ceci, S. J., Halpern, D. E., Loehlin, J. C., Perloff, R., Sternberg, R. J., & Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51 (2), 77–101. https://doi.org/10.1037/0003-066X.51.2.77
74. Nieminen, J. H., & Carless, D. (2022). Feedback literacy: a critical review of an emerging concept. Higher Education, https://doi.org/10.1007/s10734-022-00895-9
75. Panadero, E., & Lipnevich, A. A. (2022). A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review, 35, 100416. https://doi.org/10.1016/j.edurev.2021.100416
76. Pat-El, R. J., Tillema, H., Segers, M., & Vedder, P. (2013). Validation of Assessment for Learning Questionnaires for teachers and students. The British Journal of Educational Psychology, 83 (Pt 1), 98–113. https://doi.org/10.1111/j.2044-8279.2011.02057.x
77. Philips, D. C., & Burbules, N. C. (2000). Postpositivism and educational research. Rowman & Littlefield Publishers.
78. Ramaprasad, A. (1983). On the definition of feedback. Behavioral Science, 28 (1), 4–13. https://doi.org/10.1002/bs.3830280103
79. Richardson, J. T. E. (2005). Instruments for obtaining student feedback: A review of the literature. Assessment & Evaluation in Higher Education, 30 (4), 387–415. https://doi.org/10.1080/02602930500099193
80. *Rui, Y., & Muthukrishnan, P. (2019). Growth mindset and students’ perception of their English Language teachers’ feedback as predictors of language proficiency of the EFL learners. Asian EFL Journal, 23 (3), 32–60. https://www.asian-efl-journal.com/main-editions-new/2019-main-journal/volume-23-issue-3-2-2019/index.htm
81. Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. The American Psychologist, 54 (3), 182–203. https://doi.org/10.1037/0003-066X.54.3.182
82. Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75 (3), 417–453. https://doi.org/10.3102/00346543075003417
83. Stahlschmidt, S., & Stephen, D. (2020). Comparison of Web of Science, Scopus and Dimensions databases (KB Forschungspoolprojekt 2020). https://bibliometrie.info/downloads/DZHW-Comparison-DIM-SCP-WOS.PDF
84. Steelman, L. A., Levy, P. E., & Snell, A. F. (2004). The feedback environment scale: Construct definition, measurement, and validation. Educational and Psychological Measurement, 64 (1), 165–184. https://doi.org/10.1177/0013164403258440
85. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4 (1), 25–29. https://doi.org/10.1037/h0071663
86. Van der Kleij, F. M., & Lipnevich, A. A. (2021). Student perceptions of assessment feedback: a critical scoping review and call for research. Educational Assessment, Evaluation and Accountability, 33 (2), 345–373. https://doi.org/10.1007/s11092-020-09331-x
87. van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. Academic Press.
88. Vogl, E., & Pekrun, R. (2016). Emotions that matter to achievement: Student feelings about assessment. In G. T. L. Brown & L. R. Harris (Eds.), Handbook of human and social conditions in assessment (pp. 111–128). Routledge.
89. Williams, J., & Kane, D. (2008). Exploring the national student survey: Assessment and feedback issues. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=c38acc136c17c130a64845c31cef1d2c42385491
90. Wise, S. L., & Cotten, M. R. (2009). Test-taking effort and score validity: The influence of student conceptions of assessment. In D. M. McInerney, G. T. L. Brown, & G. A. D. Liem (Eds.), Student perspectives on assessment: What students can tell us about assessment for learning (pp. 187–205). Information Age Publishing.
91. Wisniewski, B., Zierer, K., & Hattie, J. (2019). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087. https://doi.org/10.3389/fpsyg.2019.03087
92. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research & Evaluation, 12 (3), 1–26. https://doi.org/10.7275/mhqa-cd89
93. Yan, Z., & Brown, G. T. L. (2017). A cyclical self-assessment process: Towards a model of how students engage in self-assessment. Assessment & Evaluation in Higher Education, 42 (8), 1247–1262. https://doi.org/10.1080/02602938.2016.1260091
94. Yan, Z., Brown, G. T. L., Lee, J. C.-K., & Qiu, X.-L. (2020). Student self-assessment: Why do they do it? Educational Psychology, 40 (4), 509–532. https://doi.org/10.1080/01443410.2019.1672038
95. Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M., … Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573 (7774), 364–369. https://doi.org/10.1038/s41586-019-1466-y
96. * Zhu, J., & Mok, M. M. C. (2018). Predicting primary students’ self-regulated learning by their prior achievement, interest, personal best goal orientation and teacher feedback. Educational Psychology, 38 (9), 1106–1128. https://doi.org/10.1080/01443410.2018.1497775
97. Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41 (2), 64–70. https://doi.org/10.1207/s15430421tip4102_2
98. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. Lissitz (Ed.), The concept of validity (pp. 65–82). Information Age Publishers.