Cognitive processing of self-report items in educational research: Do they think what we mean?

Educational Psychologist

Volumn 42, Issue 3, 2007, Pages 139-151

  Karabenick, Stuart A ^a   Woolley, Michael E ^a   Friedel, Jeanne M ^a   Ammon, Bridget V ^a   Blazevski, Julianne ^a   Bonney, Christina Rhee ^a   De Groot, Elizabeth ^a   Gilbert, Melissa C ^a   Musu, Lauren ^a   Kempler, Toni M ^b   Kelly, Kristin L ^c  

^a UNIVERSITY OF MICHIGAN   (United States)

^b RUTGERS UNIVERSITY   (United States)

^c UNIVERSITY OF TOLEDO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34548604871     PISSN: 00461520     EISSN: None     Source Type: Journal    
DOI: 10.1080/00461520701416231     Document Type: Review

References (79)

1. Ainley, M., & Hidi, S. (2002). Dynamic measures for studying interest and learning. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 43-76). Amsterdam: Elsevier.
2. Alwin, D. F. (2001). Research on survey quality. Sociological Methods and Research, 20, 1-39.
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
5. Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271.
6. Arthur, M. W., Hawkins, J. D., Pollard, J. A., Catalano, R. F., & Baglioni, A. J. (2002). Measuring risk and protective factors for substance abuse, delinquency, and other adolescent problem behaviors: The Communities That Care Youth Survey. Evaluation Review, 26, 575-601.
7. Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W. H. Freeman.
8. Barone, D. F., Maddux, J. E., & Snyder, C. R. (1997). Social cognitive psychology: History and current domains. New York: Plenum.
9. Bjorner, J. B., Kosinski, M., & Ware Jr., J. E. (2003). Calibration of an item pool for assessing the burden of headaches: An application of item response theory to the Headache Impact Test (HIT™). Quality of Life Research, 12, 913-933.
10. Blazevski, J., De Groot, E., Friedel, J., & Musu, L. (2006, April). Lessons learned from cognitive pretesting: Students 'sense of efficacy in mathematics. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
11. Boekaerts, M. (2002). The on-line motivation questionnaire: A self-report instrument to assess students' context sensitivity. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 77-120). Amsterdam: Elsevier.
12. Bong, M., & Skaalvik, E. M. (2003). Academic self-concept and self-efficacy: How different are they really? Educational Psychology Review, 15, 1-40.
13. Camara, W. J., & Lane, S. (2006). A historical perspective and current views on the standards for educational and psychological testing. Educational Measurement: Issues and Practice, 25(3), 35-41.
14. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education.
15. DeMaio, T. J., & Rothgeb, J. M. (1996). Cognitive interviewing techniques: In the lab and in the field. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for cognitive and communicative processes in survey research (pp. 177-196). San Francisco: Jossey-Bass.
16. Desimone, L. M., & Le Floch, K. C. (2004). Are we asking the right questions: Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 1-22.
17. Eccles, J. S., Wigfield, A., & Schiefele, U. (1998). Motivation to succeed. In W. Damon (Series Ed.) & N. Eisenberg (Vol. Ed.), Handbook of child psychology (5th ed.; Vol. 3: Social, emotional, and personality development; pp. 1017-1095). New York: Wiley.
18. Elliot, A. J. (2005). A conceptual history of the achievement goal construct. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 52-72). New York: Guilford.
19. Foddy, W. (1998). An empirical evaluation of in-depth probes used to pretest survey questions. Sociological Methods and Research, 27, 103-133.
20. Forsyth, B. H., & Lessler, J. T. (1991). Cognitive laboratory methods: A taxonomy. In P.P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowietz, & S. Sudman (Eds.), Measurement errors in surveys (pp. 393-418). New York: Wiley.
21. Fraser, B. (1997). Classroom environments. In H. J. H. Walberg & D. Geneva (Ed.), Psychology and educational practice (pp. 323-341). Berkeley, CA: McCutchan.
22. Gilbert, M. C., & Kempler, T. M. (2006, April). Lessons learned from cognitive pretesting: Students ' Perceptions of their teachers ' use of "real world" connections. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
23. Green, J. L., Camilli, G., & Elmore, P. B. (2006). Handbook of complementary methods in education research. Mahwah, NJ: Lawrence Erlbaum Associates.
24. Greeno, J. G. (2006). Theoretical and practical advances through research on learning. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 795-822). Mahwah, NJ: Lawrence Erlbaum Associates.
25. Hassin, R., Uleman, J. S., & Bargh, J. A. (2005). The new unconscious. New York: Oxford University Press.
26. Hastie, R. (1987). Information processing theory for survey researchers. In H. J. Hippler, N. Schwartz, & S. Sudman (Eds.), Social information processing and survey methodology (pp. 42-70). New York: Springer-Verlag.
27. Hofer, B., & Pintrich, P. R. (Eds.). (2002). Personal epistemology: The psychology of beliefs about knowledge and knowing. Mahwah, NJ: Lawrence Erlbaum Associates.
28. Jabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (1984). Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, DC: National Academy Press.
29. Järvelä, S., Salonon, P., & Lepola, J. (2002). Dynamic assessment as a key to understanding student motivation in a classroom context. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 207-240). Amsterdam: Elsevier.
30. Jobe, J. B., & Mingay, D. J. (1989). Cognitive research improves questionnaires. American Journal of Public Health, 79, 1053-1055.
31. Kane, M. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 521-535.
32. Kane, M. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38, 319-342.
33. Kaplan, A., Middleton, M. J., Urdan, T., & Midgley, C. (2002). Achievement goals and goal structures. In C. Midgley (Ed.), Goals, goal structures and patterns of adaptive learning (pp. 21-53). Mahwah, NJ: Lawrence Erlbaum Associates.
34. Karabenick, S. A. (Chair). (2006, April). Cognitive validity and the construct validation of motivation-related assessments: what, why, and how. Symposium presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
35. Karabenick, S. A., & Woolley, M. E. (2006, April). The role of cognitive validity in the construct validation of motivation-related assessment. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
36. Kelly, K. L., Bonney, C. R., & Ammon, B. V. (2006, April). Cognitive pretesting: A revealing look at students ' understanding about classroom mastery goal structure. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
37. Meece, J. L., Anderman, E. M., & Anderman, L. H. (2006). Classroom goal structure, student motivation, and academic achievement. In S. T. Fiske, A. E. Kazdin, & D. L. Schachter (Eds.), Annual review of psychology (Vol. 57, pp. 487-503), Palo Alto, CA: Annual Reviews.
38. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Education measurement (3rd ed., pp. 13-103), New York: Macmillan.
39. Messick, S, (1995), Validity of psychological assessment: Validation of inferences from persons' responses and performance as scientific inquiry into scoring meaning. American Psychologist, 9, 741-749.
40. Messick, S. (1996a). Standards-based score interpretation: Establishing valid grounds for valid inferences (Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments). Washington, DC: U.S. Government Printing Office.
41. Messick, S. (1996b). Validity of performance assessment. In G. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 1-18). Washington, DC: National Center for Educational Statistics.
42. Midgley, C., Maehr, M. L., Hicks, L., Roeser, R., Urdan, T., Anderman, E., et al. (1996). The Patterns of Adaptive Learning Scales (PALS). Ann Arbor: The University of Michigan.
43. Midgley, C., Maehr, M. L., Hruda, L., Anderman, E., Anderman, L., Freeman, K., et al. (2000). Manual for the Patterns of Adaptive Learning Scales (PALS). Ann Arbor: University of Michigan.
44. Murphy, P. K., & Alexander, P. A. (2000). A motivated exploration of motivation terminology. Contemporary Educational Psychology, 25, 3-53.
45. National Center for Education Statistics (NCES). (2003). NAEP background questionnaires. Retrieved February 8, 2005, from http://nces.ed.gov/nationsreportcard/bgquest.asp
46. National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.
47. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author.
48. Nolen, S. B. (2003). Learning environment, motivation, and achievement in high school science. Journal of Research in Science Teaching, 40, 347-368.
49. Patrick, H. (2004). Re-examining classroom mastery goal structure. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation. Volume 13: Motivating students, improving schools: The legacy of Carol Midgley (pp. 233-263). Amsterdam: Elsevier-JAI.
50. Patrick, H. (2005). Teacher practices that communicate mastery goal structure: Students ' views. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
51. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know. Washington, DC: National Academy Press.
52. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning contexts. Journal of Educational Psychology, 95, 667-686.
53. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33-40.
54. Pintrich, P. R., & Maehr, M. L. (Eds.). (2002). Advances in motivation and achievement: New directions in measures and methods (Vol. 12). Amsterdam: JAI-Elsevier.
55. Race, K. H., & Powell, K. R. (2000). Assessing student perceptions of classroom methods and activities in the context of an outcomes-based evaluation. Evaluation Review, 24, 635-646.
56. Ruiz-Prima, M. A., Schultz, S. E., Ki, M., & Shavelson, R. (2001). On the validity of cognitive interpretations of scores from alternative concept-mapping techniques. Educational Assessment, 7, 99-141.
57. Ryan, A. M., & Patrick, H. (2001). The classroom social environment and changes in adolescents' motivation and engagement during middle school. American Educational Research Journal, 38, 437-460.
58. Schwarz, N. (1999). Self-reports. American Psychologist, 54, 93-105.
59. Shrout, P. E., & Fleiss, J. L. ( 1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 86, 420-428.
60. Singer, D. G., & Revenson, T. A. (1997). A Piaget primer: How a child thinks. New York: Plume.
61. Sirken, M. G., Herrmann, D. J., Schechter, S., Schwarz, N., Tanur, J. M., Tourangeau, R. (Eds). (1999). Cognition and survey research. New York: John Wiley & Sons.
62. Stipek, D. J., Salmon, J. M., Givvin, K. B., Kazemi, E., Saxe, G., & Mac-Gyvers, V L. (1998). The value (and convergence) of practices suggested by motivation research and promoted by mathematics education reformers. Journal for Research in Mathematics Education, 29, 465-488.
63. Sudman, S., Bradburn, N., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
64. Thelk, A. D., & Hoole, E. R. (2006). What are you thinking? Postsecondary student think-alouds of scientific and quantitative reasoning items. The Journal of General Education, 55, 17-39.
65. Third International Mathematics and Science Study (TIMSS). (1995), Retrieved July 1, 2007, from http://timss.bc.edu/timss1995.html
66. Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement. International Journal of Public Opinion Research, 15, 3-7.
67. Tourangeau, R., & Rasinski, K. A. (1988). Underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.
68. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
69. Trochim, M. K. (2006). Online research methods knowledge base. Retrieved on November 1, 2006, from http://www.socialresearchmethods. net/kb/measval.htm
70. Urdan, T. (2004). Using multiple methods to assess students' perceptions of classroom goal structures. European Psychologist, 9, 222-231.
71. Urdan, T. C., Kneisel, L., & Mason, V. (1999). Interpreting messages about motivation in the classroom. In T. C. Urdan (Ed.), Advances in motivation and achievement: The role of context (Vol. 11, pp. 123-158). Stamford, CT: JAI Press.
72. Urdan, T., & Mestas, M. (2006). The goals behind performance goals. Journal of Educational Psychology, 98, 354-365.
73. Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
74. Wilson, T. D. (2002). Strangers to ourselves: Discovering the adaptive unconscious. Cambridge, MA: Harvard University Press.
75. Winne, P., Jamieson-Noel, D., & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 121-156). Amsterdam: JAI-Elsevier.
76. Winne, P., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531-566). San Diego, CA: Academic.
77. Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2004). Cognitive pretesting and the developmental validity of child self-report instruments: Theory and applications. Research on Social Work Practice, 14, 191-200.
78. Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2006), The development and evaluation of procedures to assess child self-report item validity. Educational and Psychological Measurement, 66, 687-700.
79. Yu, S. L., & Wolters, C. A. (2002). Issues in the assessment of motivation in students from ethnic minority populations. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 349-380). Amsterdam: JAI-Elsevier.