Generalizability of cognitive interview-based measures across cultural groups

Educational Measurement: Issues and Practice

Volumn 28, Issue 2, 2009, Pages 9-18

  Solano Flores, Guillermo ^a   Li, Min ^b  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Assessment; Cognitive interviews; Cognitive validity; Culture; Generalizability theory

Indexed keywords

EID: 66749185844     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2009.00143.x     Document Type: Article

References (74)

1. Allalouf, A., Hambleton, R., Sireci, S. (1999). Identifying the causes of DIF on verbal items. Journal of Educational Measurement, 36, 18 198.
2. Ayala, C. C., Shavelson, R., Yin, Y., Schultz, S. E. (2002). Reasoning dimensions underlying achievement: The case of performance assessment. Educational Assessment, 8 (2 101 121. (Pubitemid 36543729)
3. Baxter, G. P., Elder, A. D., Glaser, R. (1996). Knowledge-based cognition and performance assessment in the science classroom. Educational Psychologist, 31 (2 133 140.
4. Baxter, G. P., Glaser, R. (1998). Investigating the cognitive complexity of science assessments. Educational Measurement: Issues and Practice, 17 (3 37 45.
5. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds. 1999). How people learn: Brain, mind, experience, and school. Washington, DC : National Academy Press.
6. Brennan, R. L. (2007). Unbiased estimates of variance components with Bootstrap procedures. Educational and Psychological Measurement, 67, 784 803.
7. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA : The American College Testing Program.
8. Brennan, R. L. (2001). Generalizability theory. New York : Springer-Verlag.
9. Brennan, R. L., Gao, X., Colton, D. A. (1995). Generalizability analyses of Work Keys listening and writing tests. Educational and Psychological Measurement, 55 (2 157 176.
10. Brennan, R. L., Johnson, E. G. (1995). Generalizability of performance assessments. Educational Measurement: Issues and Practice, 14 (4 9 12.
11. Brenner, M. E. (1998). Meaning and money. Educational Studies in Mathematics, 36, 123 155. (Pubitemid 128136583)
12. Chi, M. T. H., Feltovich, P. G., Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121 152.
13. Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds. 1988). The nature of expertise. Hillsdale, NJ : Lawrence Erlbaum.
14. Cronbach, L. J., Gleser, G. C., Nanda, H., Rajaratnam, N. (1972). The dependability of behavioral measurements. New York : Wiley.
15. Crystal, D. (1997). The Cambridge encyclopedia of language (2nd ed.). Cambridge : Cambridge University Press.
16. Delpit, L. (1995). Other people's children: Cultural conflict in the classroom. New York : New Press.
17. Drasgow, F., Porbst, T. M. (2005). The psychometrics of adaptation: Evaluating measurement equivalence across languages and cultures. In R. Hambleton, P. Merenda, & C. D. Spielberger (Eds. Adapting educational and psychological tests for cross-cultural assessment (pp. 265 296). Mahwah, NJ : Lawrence Erlbaum.
18. Ericsson, K. A. (2005). Recent advances in expertise research: A commentary on the contributions to the special issue. Applied Cognitive Psychology, 19, 233 241. (Pubitemid 40394184)
19. Ericsson K. A., Simon, H. A. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge, MA : MIT Press.
20. Estrin, E., Nelson-Barber, S. (1995). Issues in cross-cultural assessment: American Indian and Alaska native students. Knowledge Brief # 12. San Francisco : Far West Laboratory.
21. Ferrara, S., García-Duncan, T., Freed, R., Vélez-Paschke, A., McGivern, J., Mushlin, S., Mattessich, A., Rogers, A., Westphalen, K. (2004). Examining test score validity by examining item construct validity: Preliminary analysis of evidence of the alignment of targeted and observed content, skills, and cognitive processes in a middle school science assessment. In E. A. Vanderputten (Chair Putting Alignment to the Test. Symposium conducted at the American Educational Research Association, April, San Diego, CA.
22. Fisher, R. P., Geiselman, R. E. (1992). Memory-enhancing techniques for the investigative interviewing. Springfield, IL : Charles C. Thomas.
23. Gao, X., Shavelson, R. J., Baxter, G. P. (1994). Generalizability of large-scale performance assessments in science: Promises and problems. Applied Measurement in Education, 7, 323 342.
24. Greenfield, P. M. (1997). Culture as process: Empirical methods for cultural psychology. In J. W. Berry, Y. H. Poortinga, & J. Pandey (Eds. Handbook of cross-cultural psychology: Theory and method (2nd ed., Vol. 1, pp. 301 346). Needham Heights, MA : Allyn & Bacon.
25. Hambleton, R. K., Swaminathan, H., Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA : Sage.
26. Hamilton, L. S., Nussbaum, E. M., Snow, R. E. (1997). Interview procedures for validating science assessments. Applied Measurement in Education, 10, 181 200. (Pubitemid 127471652)
27. Heath, S. B. (1983). Ways with words: Language, life, and work in communities and classrooms. New York : Cambridge University Press.
28. Hively, W., Patterson, H. L., Page, S. H. (1968). A "universe- defined" system of arithmetic achievement tests. Journal of Educational Measurement, 5 (4 275 290.
29. Huberman, M., Levine, R. (1999). Validation of fourth and eighth grade students' responses to reading, math, and science NAEP background items: Use of cognitive interviewing techniques with teachers and students. Washington, DC : American Institutes for Research Report.
30. Kane, M. T. (1982). A sampling model for validity. Applied Psychological Measurement, 6, 125 160.
31. Karabenick, S. A., Woolley, M. E., Gilbert, M. C., Kempler, T. M., Blazevksi, J., DeGroot, E., et al 2006). Cognitive validity and the construct validation of motivation-related assessments: What, why, and how. Paper presented at the Annual American Educational Research Association Meeting, San Francisco, CA.
32. Kazemi, E. (2002). Exploring test performance in mathematics: The questions children's answers raise. Journal of Mathematical Behavior, 21, 203 224.
33. Kraiger, K., Teachout, M. S. (1990). Generalizability theory as construct-related evidence for the validity of job performance ratings. Human Performance, 3, 19 35.
34. Kusimo, P., Ritter, M. G., Busick, K., Ferguson, C., Trumbull, E., Solano-Flores, G. (2000). Making Assessment Work for Everyone: How to Build on Student Strengths. Regional Educational Laboratories (http://www.sedl.org/pubs/ tl05/ accessed February 25, 2008).
35. Lane, S., Liu, M., Ankenmann, R. D., Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33 (1 71 92.
36. Larkin, J., McDermott, J., Simon, D. P., Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335 1342.
37. Lee, O. (2002). Promoting scientific inquiry with elementary students from diverse cultures and languages. Review of Research in Education, 26, 23 69.
38. Li, D., Brennan, R. (2007). A multi-group generalizability analysis of a large-scale reading comprehension test. Paper presented at the Annual Meeting of the National Council on Measurement in Education. Chicago, IL, April 10-12.
39. Li, M., Solano-Flores, G., Kwon, M., Tsai, S. P. (2008, April It's asking me as if I were the mother:" Examining how students from different groups interpret test items. Paper presented at the National Association for Research in Science Teaching Annual Meeting, Baltimore, MD.
40. Luykx, A., Lee, O., Mahotiere, M., Lester, B., Hart, J., Deaktor, R. (2007). Cultural and home language influences on children's responses to science assessments. Teachers College Record, 109 (4 897 926.
41. Megone, M. E., Cai, J., Silver, E. A., Wang, N. (1994). Validating the cognitive complexity and content quality of a mathematics performance assessment. International Journal of Educational Research, 21 (3 317 340.
42. National Assessment of Educational Progress (1996). Mathematics items public release. Washington, DC : Author.
43. Nuthall, G., Alton-Lee, A. (1995). Assessing classroom learning: How students use their knowledge and experience to answer classroom achievement test questions in science and social studies. American Educational Research Journal, 2 (1 185 223.
44. Pellegrino, J. W., Chudowsky, N., Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC : National Academy Press.
45. Resnick, L. B. (1985). Cognition and instruction: Recent theories of human competence and how it is acquired. In B. L. Hammonds (Ed. Psychology and learning: The master lecture series (Vol. 4, pp. 123 186). Washington, DC : American Psychological Association.
46. Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New York : Oxford University Press.
47. Rogoff, B. (1995). Observing sociocultural activity on three planes: Participatory appropriation, guided participation, and apprenticeship. In J. V. Wertsch, P. del Río, & A. Alvarez (Eds. Sociocultural studies of mind. New York : Cambridge University Press.
48. Ruiz-Primo, M. A., Baxter, G. P., Shavelson, R. J. (1993). On the stability of performance assessments. Journal of Educational Measurement, 30, 41 53.
49. Ruiz-Primo, M. A., Shavelson, R. J., Li, M., Schultz, S. E. (2001). On the validity of cognitive interpretations of scores from alternative mapping techniques. Educational Assessment, 7 (2 99 141.
50. Schoenfeld, A. H. (1991). On mathematics sense making: An informal attack on the unfortunate divorce of formal and informal mathematics. In J. F. Voss, D. N. Perkins, & J. W. Segal (Eds. Informal reasoning and education (pp. 311 343). Hillsdale, NJ : Lawrence Erlbaum Associates.
51. Schoenfeld, A. H. (2004). The math wars. Educational Policy, 18 (1 253 286. (Pubitemid 38157515)
52. Shavelson, R. J., Baxter, G. P., Gao, X. (1993). Sampling variability of performance assessment. Journal of Educational Measurement, 30 (3 215 232.
53. Shavelson, R. J., Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA : Sage.
54. Simon, D. P., Simon, H. A. (1978). Individual differences in solving physics problems. In R. S. Siegler Ed. Children's thinking: What develops pp. 325 348). Hillsdale, NJ : Erlbaum.
55. Solano-Flores, G. (2001). World views and test views: The relevance of cultural validity. Paper presented at the European Association of Research in Learning and Instruction. Fribourg, Switzerland, August 28-September 1.
56. Solano-Flores, G. (2005). Examining test item meaningfulness across cultural groups. Paper presented at the 11th Biennial Conference of the European Association for Research on Learning and Instruction. Nicosia, Cyprus, August 23-27.
57. Solano-Flores, G. (2006). Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English-language learners. Teachers College Record, 108 (11 2354 2379.
58. Solano-Flores, G., Li, M. (2006). The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice, 25 (1 13 22.
59. Solano-Flores, G., Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38 (5 553 573. (Pubitemid 33655580)
60. Solano-Flores, G., Raymond, B., Schneider, S. A., Timms, M. (1999). Management of scoring sessions in alternative assessment: The computer-assisted scoring approach. Computers and Education, 33, 47 63. (Pubitemid 129623592)
61. Solano-Flores, G., Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32 (2 3 13. (Pubitemid 36412722)
62. Stein, L. M., Memon, A. (2006). Testing the efficacy of the cognitive interview in a developing country. Applied Cognitive Psychology, 20, 597 605. (Pubitemid 44082761)
63. Stockwell, P. (2002). Sociolinguistics: A resource book for students. New York : Routledge.
64. The University of Iowa Center for Advanced Studies in Measurement and Assessment (2004). Computer Programs (http://www.education.uiowa.edu/casma/ computer-programs.htm).
65. Tong, Y., Brennan, R. L. (2007). Bootstrap estimates of standard errors in Generalizability theory. Educational and Psychological Measurement, 67, 804 817.
66. Van de Vijver, F., Leung, K. (1997). Methods and data analysis of comparative research. In J. W. Berry, Y. H. Poortinga, & J. Pandey (Eds. Handbook of cross-cultural psychology: Theory and method (2nd ed., Vol. 1). Needham Heights, MA : Allyn & Bacon.
67. van Geert, P., van Dijk, M. (2003). Ambiguity in child language: The problem of interobserver reliability in ambiguous observation data. First Language, 23 (3 259 284.
68. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA : Harvard University Press.
69. Wardhaugh, R. (2002). An introduction to sociolinguistics (4th ed.). Malden, MA : Blackwell Publishing.
70. Warren, B., Ballenger, C., Ognowski, M., Rosebery, A. S., Hudicourt-Barnes, J. (2001). Rethinking diversity in learning science: The logic of everyday sense-making. Journal of Research in Science Teaching, 38 (5 529 552. (Pubitemid 33655579)
71. Welzel, M., Roth, W. M. (1998). Do interviews really assess students' knowledge International Journal of Science Education, 20 (1 25 44. (Pubitemid 128691878)
72. Wertsch, J. V. (1985). Vygotsky and the social formation of mind. Cambridge, MA : Harvard University Press.
73. Wertsch, J. V. (1998). Mind as action. New York : Oxford University Press.
74. Willis, G. B., Royston, P., Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires. Applied Cognitive Psychology, 5, 251 267.