Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations

Journal of Educational Measurement

Volumn 40, Issue 2, 2003, Pages 163-184

  Rodriguez, Michael C ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0037697027     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2003.tb01102.x     Document Type: Article

References (79)

1. Ackerman, T. A., & Smith, P. L. (1988). A comparison of the information provided by essay, multiple-choice, and free-response writing tests. Applied Psychological Measurement, 12(2), 117-128.
2. Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole Publishing Co.
3. Barnett-Foster, D., & Nagy, P. (1996). Undergraduate student response strategies to test questions of varying format. Higher Education, 32(2), 177-198.
4. Bennett, R. E., Rock, D. A., Braun, H. I., Frye, D., Spohrer, J. C., & Soloway, E. (1990). The relationship of expert-system score constrained free-response items to multiple-choice and open-ended items. Applied Psychological Measurement, 14, 151-162.
5. Bennett, R. E., Rock, D. A., & Wang, M. (1991). Equivalence of free-response and multiplechoice items. Journal of Educational Measurement, 28(1), 77-92.
6. Bennett, R. E., & Ward, W. C. (Eds.). (1993). Construction versus choice in cognitive measurement. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
7. Berg, C. A., & Smith, P. (1994). Assessing students' abilities to construct and interpret line graphs: Disparities between multiple-choice and free-response instruments. Science Education, 78(6), 527-554.
8. Birenbaum, M., & Tatsuoka, K. K. (1987). Open-ended versus multiple-choice response formats - It does make a difference for diagnostic purposes. Applied Psychological Measurement, 11, 385-395.
9. Birenbaum, M, Tatsuoka, K. K., & Gutvirtz, Y. (1992). Effects of response format on diagnostic assessment of scholastic achievement. Applied Psychological Measurement, 76(4), 353-363.
10. Bracht, G. H., & Hopkins, K. D. (1970). The communality of essay and objective tests of academic achievement. Educational and Psychological Measurement, 30, 359-364.
11. Breland, H.M., Danos, D.O., Kahn, H.D., Kubota, M.Y., & Bonner, M.W. (1994). Performance versus objective testing and gender: An exploratory study of an advanced placement History examination. Journal of Educational Measurement, 31(4), 275-293.
12. Breland, H. M., & Gaynor, G. L. (1979). A comparison of direct and indirect assessments of writing skill. Journal of Educational Measurement, 16(2), 119-128.
13. Bridgeman, B. (1992). A comparison of quantitative questions in open-ended and multiplechoice formats. Journal of Educational Measurement, 29, 253-271.
14. Bridgeman, B., Hale, G. A., Lewis, C., Pollack, J., & Wang, M. (1992). Placement validity of a prototype SAT with an essay (Research Rep. 92-28). Princeton, NJ: Educational Testing Service.
15. Bridgeman, B., & Lewis, C. (1994). The relationship of essay and multiple-choice scores with college courses. Journal of Educational Measurement, 31, 37-50.
16. Bridgeman, B., & Rock, D. A. (1993). Relationship among multiple-choice and open-ended analytical questions. Journal of Educational Measurement, 30, 313-329.
17. Carcelli, L., Taylor, C., & White, K. (1980). The effect of item format on phonics subtest scores of standardized reading achievement tests. Logan, UT: Utah State University. (ERIC Document Reproduction Service No. ED206654)
18. Cirn, J. T. (1986). True/false versus short answer questions. College Teaching, 34, 34-37.
19. Coffman, W. E. (1966). On the validity of essay tests of achievement. Journal of Educational Measurement, 3, 151-156.
20. Coulson, J. E., & Silberman, H. F. (1960). Effects of three variables in a teaching machine. Journal of Educational Psychology, 51(5), 135-143.
21. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich, Inc.
22. Davis, F. B., & Fifer, G. (1959). The effect on test reliability and validity of scoring aptitude and achievement tests with weights for every choice. Educational and Psychological Measurement, 14(2), 159-170.
23. DeMars, C. (1998). The impact of test consequences and response format on performance. Unpublished doctoral dissertation, Michigan State University.
24. Downing, S. M., & Norcini, J. J. (1998, April). Constructed response or multiple choice: Does the format make a difference for prediction? Paper presented at the annual meeting of the American Educational Research Association, San Diego.
25. Ercikan, K., & Schwarz, R. (1995, April). Dimensionality of multiple-choice and constructed-response tests for different ability groups. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
26. Eurich, A.C. (1931). Four types of examinations compared and evaluated. Journal of Educational Psychology, 26, 268-278.
27. Fisher, G. L. (1996). The validity of pre-calculus multiple-choice and performance-based testing as a predictor of undergraduate mathematics and chemistry achievement. Unpublished master's thesis, University of California, Santa Barbara.
28. Frary, R. B. (1985). Multiple-choice versus free response: A simulation study. Journal of Educational Measurement, 22, 21-31.
29. Frisbie, D. A., & Cantor, N. K. (1995). The validity of scores from alternative methods of assessing spelling achievement. Journal of Educational Measurement, 32(1), 55-78.
30. Gay, L. (1980). The comparative effects of multiple-choice versus open-ended on retention. Journal of Educational Measurement, 17, 45-50.
31. Godshalk, F. I., Swineford, F., & Coffman, W. E. (1966). The measurement of writing ability. New York: College Entrance Examination Board.
32. Haladyna, T. M. (1998, April). Fidelity and proximity to criterion: When should we use multiple-choice? Paper presented at the annual meeting of the American Educational Research Association, San Diego.
33. Halpin, G. (1979, April). Retention in an actual classroom setting as a function of type and complexity of tests. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
34. Hancock, G. R. (1992). Cognitive complexity and the comparability of multiple-choice and constructed-response test formats. Journal of Experimental Education, 62(2), 143-157.
35. Harke, D. J., Herron, J. D., & Leffler, R. W. (1972). Comparison of a randomized multiplechoice format with a written on-hour physics problem test. Science Education, 56, 563-565.
36. Hedges, L. V. (1994). Statistical considerations. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 29-38). New York: Russell Sage Foundation.
37. Heim, A. W., & Watts, K. P. (1967). An experiment on multiple-choice versus open-ended answering in a vocabulary test. British Journal of Educational Psychology, 37, 339-346.
38. Hogan, T. P., & Mishler, C. (1980). Relationships between essay tests and objective tests of language skills for elementary school students. Journal of Educational Measurement, 77(3), 219-227.
39. Horn, J. L. (1966). Some characteristics of classroom examinations. Journal of Educational Measurement, 3, 292-295.
40. Hurd, A. W. (1932). Comparison of short answer and multiple-choice tests covering identical subject content. Journal of Educational Research, 26, 28-30.
41. Hurlbut, D. (1954). The relative value of recall and recognition techniques for measuring precise knowledge of word meaning - nouns, verbs, adjectives. Journal of Educational Research, 47, 561-576.
42. Kumar, V. K., Robinsky, L., & Pandey, T. N. (1979). Test mode, test instruction, and retention. Contemporary Educational Psychology, 4, 211-218.
43. Loyd, B. H., & Steele, J. (1986). Assessment of reading comprehension: A comparison of constructs. Reading Psychology, 7, 1-10.
44. Lukhele, R., Thissen, D., & Wainer, H. (1994). On the relative value of multiple-choice, constructed-response, and examinee-selected items on two achievement tests. Journal of Educational Measurement, 30, 234-250.
45. Magill, W. H. (1934). The influence of the form of item on the validity of achievement tests. Journal of Educational Psychology, 25, 21-28.
46. Manhart, J. J. (1996, April). Factor analytic methods for determining whether multiplechoice and constructed-response tests measure the same construct. Paper presented at the annual meeting of the American Educational Research Association, New York.
47. Martinez, M. E. (1991). A comparison of multiple-choice and constructed response figurai response items. Journal of Educational Measurement, 28, 131-145.
48. Martinez, M.E. (1998, April). Cognition and the question of test item format. Paper presented at the annual meeting of the American Educational Research Association, San Diego.
49. Messick, S. (1993). Trait equivalence as construct validity of score interpretation across multiple methods of measurement. In R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement (pp. 61-74). Hillsdaie, NJ: Lawrence Erlbaum Associates, Inc.
50. Millman, J., & Setijadi. (1966). A comparison of performance of American and Indonesian students on three types of test items. The Journal of Educational Research, 59(6), 273-275.
51. Morgan, R., & Manackshana, B. (1996, April). The psychometric perspective: Meeting four decades of challenges. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
52. Moss, P. A., Cole, N. S., & Khampalikit, C. (1982). A comparison of procedures to assess written language skills at Grades 4, 7, and 10. Journal of Educational Measurement, 19(1), 37-47.
53. Oosterhof, A.C., & Coats, P. K. (1984). Comparison of difficulties and reliabilities of quantitative word problems in completion and multiple-choice item formats. Applied Psychological Measurement, 8, 287-294.
54. Paterson, D.G. (1926). Do new and old type examinations measure different mental functions? School and Society, 24, 246-248.
55. Pollock, J. M. (1997). Constructed response tests in the NELS:88 high school effectiveness study. Washington, DC: National Center for Education Statistics, U.S. Department of Education, Office of Educational Research and Improvement.
56. Quellmalz, E. S., Capell, F. J., & Chou, C.-P. (1982). Effects of discourse and response mode on the measurement of writing competence. Journal of Educational Measurement, 19(4), 241-258.
57. Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage Foundation.
58. Rodriguez, M. C. (2002). Choosing an item format. In G. Tindal & T. M. Haladyna (Eds.), Large-scale assessment programs for all students: Development, implementation, and analysis (pp. 213-231). Mahwah, NJ: Lawrence Erlbaum Associates.
59. Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage Foundation.
60. Rowley, G. L. (1974). Which examinees are most favoured by the use of multiple choice tests? Journal of Educational Measurement, 11(1), 15-23.
61. Ruch, G.M., & Charles, J.W. (1928). A comparison of five types of objective tests in elementary psychology. Journal of Applied Psychology, 12, 398-403.
62. Ruch, G. M., & Stoddard, G. D. (1925). Comparative reliabilities of five types of objective examinations. Journal of Educational Psychology, 16, 89-103.
63. Ryan, J., & Franz, S. (1998, April). Gender X format interactionist perspective. Paper presented at the annual meeting of the American Educational Research Association, San Diego.
64. Sax, G., & Collet, L. S. (1968). An empirical comparison of the effects of recall and multiple-choice tests on student achievement. Journal of Educational Measurement, 5(2), 169-173.
65. Shadish, W. R., & Haddock, C. K. (1994). Combining estimates of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 261-284). New York: Russell Sage Foundation.
66. Smith, J. K., & Smith, M. R. (1984, April). The influence of item format on measures of reading comprehension. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.
67. Thiede, K. W., Klockars, A. J., & Hancock, G. R. (1991, April). Recognition versus recall test formats: A correlational analysis. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
68. Thissen, D., Wainer, H., & Wang, X.-B. (1994). Are tests comprising both multiple-choice and free-response items necessarily less unidimensional than multiple-choice tests? An analysis of two tests. Journal of Educational Measurement, 31(2), 113-123.
69. Traub, R. E. (1993). On the equivalence of the traits assessed by multiple-choice and constructed-response tests. In R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement (pp. 29-44). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
70. Traub, R. E., & Fisher, C. W. (1977). On the equivalence of constructed-response and multiple-choice tests. Applied Psychological Measurement, 1, 355-369.
71. Traub, R. E., & MacRury, K. (1990). Multiple-choice vs. free-response in the testing of scholastic achievement. Unpublished manuscript, The Ontario Institute for Studies in Education. Published in German in K. Ingenkamp & R. S. Jager (Eds.) Tests und trends 8: Jahrbuch der Padagogischen Diagnostik (pp. 128-159). Weinheim & Basel: Beltz Verlag.
72. van den Bergh, H. (1990). On the construct validity of multiple-choice items for reading comprehension. Applied Psychological Measurement, 14, 1-12.
73. Veloski, J. J., Robeson, M. R., Rovinelli, R. J., & Hojat, M. (1993). IRT calibration of open-ended and multiple-choice test questions. Paper presented at the annual meeting of the National Council on Measurement in Education, Atlanta.
74. Vernon, P. E. (1962). The determinants of reading comprehension. Educational and Psychological Measurement, 9, 430-449.
75. Wainer, H., & Thissen, D. (1993). Combining multiple-choice and constructed-response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education 6(2), 103-118.
76. Ward, W. C. (1982). A comparison of free-response and multiple-choice forms of verbal aptitude tests. Applied Psychological Measurement, 6, 1-11.
77. Ward, W. C., Dupree, D., & Carlson, S. B. (1987). A comparison of free-response and multiple-choice questions in the assessment of reading comprehension. Princeton, NJ: Educational Testing Service.
78. Ward, W. C., Frederiksen, N., & Carlson, S. B. (1980). Construct validity of free-response and machine-scorable forms of a test. Journal of Educational Measurement, 17(1), 11-29.
79. Wilson, M., & Wang, W. (1995). Complex composites: Issues that arise in combining different modes of assessment. Applied Psychological Measurement, 19(1), 51-71.