Stability of country rankings across item formats in the third international mathematics and science study

Educational Measurement: Issues and Practice

Volumn 21, Issue 4, 2002, Pages 27-38

  O'Leary, Michael ^a  

^a DUBLIN CITY UNIVERSITY   (Ireland)

Author keywords

Country rankings; International comparisons; Item format; Science assessment; TIMSS

Indexed keywords

EID: 34250362554     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2002.tb00104.x     Document Type: Article

References (75)

1. Airasian, P. W. (1997) Classroom assessment. New York: McGraw-Hill.
2. Airasian, P. W., &Madaus, G. F. (1983) Linking testing and instruction: Policy issues. Journal of Educational Measurement, 20, 103–118.
3. Aschbacker, P. R. (1991) Performance assessment: State activity, interest, and concerns. Applied Measurement in Education, 4, 275–278.
4. Atkin, J. M., &Black, P. (1997) Policy perils of international comparisons. Phi Delta Kappan, 79, 22–28.
5. Baker, D. P. (1997) Surviving TIMSS: Or everything you blissfully forgot about international comparisons. Phi Delta Kappan, 79, 295–300.
6. Beaton, A. E. (1998) Comparing cross-national student performance on TIMSS using different test items. International Journal of Educational Research, 29, 529–542.
7. Beaton, A. E., &Gonzalez, E. J. (1997) TIMSS test-curriculum matching analysis. In M. O. Martin & D. L. Kelly, (Eds.), Third International Mathematics and Science Study technical report: Vol. 2. Design and development (pp. 187–193). Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
8. Beaton, A. E., Martin, M. O., &Mullis, I. V. S. (1997) Providing data for educational policy in an international context: The Third International Mathematics and Science Study (TIMSS). European Journal of Psychological Assessment, 13, 49–58.
9. Beaton, A. E., Martin, M. O., Mullis, I. V. S., Gonzalez, E. J., Smith, T. A., &Kelly, D. L. (1996) Science achievement in the middle school years: IEA'S Third International Mathematics and Science Study. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
10. Bennet, R. E., Rock, D. A., &Wang, M. (1991) Equivalence of free response and multiple-choice items. Journal of Educational Measurement, 28, 77–92.
11. Bennet, R. E., &Ward, W. C. (Eds.). (1993) Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessments. Hillsdale, NJ: Erlbaum.
12. Birenbaum, M., &Tatsuoka, K. K. (1987) Open-ended versus multiple-choice response formats–-It does make a difference for diagnostic assessment. Applied Psychological Measurement, 11, 385–395.
13. Birenbaum, M., Tatsuoka, K. K., &Gutvirtz, Y. (1992) Effects of response format on diagnostic assessment of scholastic achievement. Applied Psychological Measurement, 16, 353–363.
14. Bolger, N., &Kellaghan, T. (1990) Method of measurement and gender differences in scholastic achievement. Journal of Educational Measurement, 27, 165–174.
15. Bridgeman, B. (1992) A comparison of quantitative questions in open-ended and multiple-choice format. Journal of Educational Measurement, 29, 253–271.
16. Choppin, B. F., &Purves, A. L. (1969) A comparison of open-ended and multiple-choice items dealing with literacy understanding. Research in the Teaching of English, 3, 15–24.
17. Colvin, R. L. (1996, November 21) Global study finds U.S. students weak in math. Los Angeles Times, p. A1.
18. Comber, L. C., &Keeves, J. P. (1973) Science education in nineteen countries: An empirical study. New York: Wiley.
19. Cooley, W. W., &Leinhart, G. (1980) The instructional dimensions study. Educational Evaluation and Policy Analysis, 2, 7–25.
20. Darling-Hammond, L. (1994) Performance assessment and educational equity. Harvard Educational Review, 64, 5–30.
21. Elley, W. B. (Ed.). (1992) How in the world do students read? The Hague, the Netherlands: International Association for the Evaluation of Educational Achievement.
22. Elley, W. B., &Mangubhai, F. (1992) Multiple-choice and open-ended items in reading tests: Same or different Studies in Educational Evaluation, 18, 191–199.
23. Ercikan, K. (1998) Translation effects in international assessments. International Journal of Educational Research, 29, 534–553.
24. Gipps, C. V. (1995) Reliability, validity, and manageability in large-scale assessment. In H. Torrance, (Ed.), Evaluating authentic assessment (pp. 105–123). Buckingham, England: Open University Press.
25. Gipps, C. V., &Murphy, P. (1994) A fair test?: Assessment, achievement, and equality. Buckingham, England: Open University Press.
26. Goldstein, H. (1997) International comparisons of student achievement. In A. Little & A. Wolf, (Eds.), Assessment in transition: Learning, monitoring and selection in international perspective (pp. 58–87). Oxford, England: Pergamon Press.
27. Guskey, T. R., &Kifer, E. W. (1990) Ranking school districts on the basis of statewide test results: Is it meaningful or misleading Educational Measurement: Issues and Practice, 9, 11–16.
28. Hambleton, R. (1992) Translation achievement tests for use in cross-national studies. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
29. Hamilton, L. S. (1997, March) Identifying differential item functioning on science achievement tests. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
30. Huchison, D., &Schagen, I. (Eds.). (1994) How reliable is national curriculum, assessment. Slough, England: NFER-Nelson.
31. International Association for the Evaluation of Educational Achievement. (1997) TIMSS international database: Middle school years 1995 assessment (CD-ROM). Chestnut Hill, MA: TIMSS International Study Center, Boston College.
32. Innerst, C. (1996, November 21.) U.S. students lag in math, science. Washington Times, p. A1.
33. Ireland. (1999) Commission on the points system. Dublin: The Stationery Office.
34. Jakwerth, P. M., Bianchi, L. J., Houang, R. T., Schmidt, W. H., Valverde, G. A., Wolfe, R. G., &Yang, W. (1997, March) Validity in cross-national assessments: Pitfalls and possibilities. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
35. Kapinus, B., &Atash, N. (1995) Exploring the possibilities of constructed-response items. In M. Binkley, K. Rust, &M. Winglee, (Eds.), Methodological issues in comparative educational studies (pp. 105–129). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
36. Kellaghan, T., &Greaney, V. (2001) The globalisation of assessment. Assessment in Education, 8, 87–102.
37. Kellaghan, T., &Grisay, A. (1995) International comparisons of student achievement: Problems and prospects. In Measuring what students learn. Paris: Organisation for Economic Cooperation and Development.
38. Kiplinger, V. L., &Linn, R. L. (19951996) Raising the stakes of test administration: The impact on student performance on the National Assessment of Educational Progress. Educational Assessment, 3, 111–133.
39. Kjoernsli, M., &Jorde, D. (1992, April) Evaluation in science: Content or process. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
40. Lally, C. (1997, August 31.) Poor literacy linked to course attendance. Irish Times, p. 8.
41. Lapointe, A. E., Askew, J. M., &Mead, N. A. (1992) Learning science. Princeton, NJ: Educational Testing Service.
42. Lukhele, R., Thissen, D., &Wainer, H. (1994) On the relative value of multiple-choice, constructed response and examinee selected items on two achievement tests. Journal of Educational Measurement, 31, 234–250.
43. Madaus, G. F., &Kellaghan, T. (1992) Curriculum evaluation and assessment. In P. W. Jackson, (Ed.), Handbook of curriculum research. New York: Macmillan.
44. Madaus, G. F., &Kellaghan, T. (1993) The British experience with authentic assessment. Phi Delta Kappan, 74, 469–485.
45. Martin, M. O., &Kelly, D. L. (Eds.). (1996) Third International Mathematics and Science Study technical report: Vol. 1. Design and development. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
46. Martin, M. O., &Kelly, D. L. (Eds.). (1997) Third International Mathematics and Science Study technical report: Vol. 2. Implementation and analysis. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
47. Martin, M. O., Mullis, I. V. S., Beaton, A. E., Gonzalez, E. J., Smith, T. A., &Kelly, D. L. (1997) Science achievement in the primary school years: IEA's Third International Mathematics and Science Study. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
48. Messick, S. (1989) Validity. In R. L. Linn, (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.
49. Messick, S. (1993) Trait equivalence as construct validity of score interpretation across multiple methods of measurement. In R.E. Bennet & W.C. Ward, (Eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessments (pp. 61–73). Hillsdale, NJ: Erlbaum.
50. Mislevy, R. J. (1995) What can we learn from international assessments Educational Evaluation and Policy Analysis, 4, 419–437.
51. Mullis, I. V. S., Martin, M. O., Beaton, A. E., Gonzalez, E. J., Kelly, D. L., &Smith, T. A. (1998) Mathematics and science achievement in the final year of secondary school. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.
52. Mullis, I. V. S., &Smith, T. A. (1996) Quality control steps for free-response scoring. In M. O. Martin & I. V. S. Mullis, (Eds.), Quality assurance in data collection (pp. 5-1–5-32). Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Public Policy, Boston College.
53. O'Leary, M. (1999) The validity and consistency of findings from international comparative studies of student achievement: A comparison of outcomes from IAEP2 and TIMSS. Unpublished doctoral dissertation, Boston College, Chestnut Hill, MA.
54. O'Leary, M. (2001) The effects of age-based and grade-based sampling on the relative standing of countries in international comparative studies of student achievement. British Journal of Educational Research, 21, 187–200.
55. O'Leary, M., Madaus, G., Kellaghan, T., &Beaton, A. (2000) Consistency of findings across international surveys of mathematics and science achievement: A comparison of IAEP2 and TIMSS. Education Policy Analysis Archives 8. Retrieved August 23, 2000, from http://epaa.asu.edu/epaav8n43.html.
56. O'Neill, H. F., Sugrue, B., &Baker, E. L. (19951996) Effects of motivational interventions on the National Assessment of Educational Progress mathematics performance. Educational Assessment, 3, 135–157.
57. Organisation for Economic Cooperation and Development. (1999) Measuring student knowledge and skills: A new framework for assessment. Paris: Author.
58. Perkhounkova, Y., Hoover, H. D., &Ankemann, R. D. (1997, March) An examination of construct validity of multiple-choice versus constructed response tests. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
59. Ramseier, E. (1997, September) Task difficulty and curricular priorities in science: Analysis of typical features of the Swiss performance in TIMSS. Paper presented at the annual meeting of the European Conference on Educational Research, Frankfurt, Germany.
60. Robitaille, D. F., &Garden, R. (Eds.). (1996) Research questions and study design (TIMSS Monograph No. 2). Vancouver, British Columbia, Canada: Pacific Educational Press.
61. Robitaille, D. F., Schmidt, W. H., Raizen, S., McKnight, C., Britton, E., &Nicol, C. (1993) Curriculum frameworks for mathematics and science (TIMSS Monograph No. 1). Vancouver, British Columbia, Canada: Pacific Educational Press.
62. Stedman, L. C. (1994) Incomplete explanations: The case of U.S. performance in the international assessments of education. Educational Researcher, 23(7), 24–32.
63. Stuart, D. (1995) Notes on carrying out the test curriculum matching analysis: June-September 1995. Unpublished manuscript.
64. Swinton, S. S. (1997, March) Block position effects as indirect indicators of motivation in the 1996 NAEP science assessment. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
65. Thissen, D., Wainer, H., &Wang, X. B. (1994) Are tests comprising both multiple-choice and free-response items necessarily less unidimensional than multiple-choice tests: An analysis of two tests. Journal of Educational Measurement, 31, 113–123.
66. TIMSS International Study Center. (1995a) Main survey test items. Chestnut Hill, MA: Author.
67. TIMSS International Study Center. (1995b) Science Population 2 test-curriculum matching analysis answer booklet. Chestnut Hill, MA: Author.
68. TIMSS International Study Center. (1995c) Science Population 2 items sorted by content grouping. Chestnut Hill, MA: Author.
69. TIMSS International Study Center. (1996) P-value almanac for cognitive items. Science Population 2: Upper grade. Chestnut Hill, MA: Author.
70. Traub, R. E., &Fisher, C. W. (1977) On the equivalence of constructed-response and multiple-choice tests. Applied Psychological Measurement, 1, 355–369.
71. Van den Berg, H. (1988) Examens Ge-examineerd [Examining the examinations]. The Hague, the Netherlands: S.V.O.
72. Vernon, P. E. (1962) The determinants of reading comprehension. Educational and Psychological Measurement, 22, 269–286.
73. Winer, B. J., Brown, D. R., &Michels, K. M. (1991) Statistical principles in experimental design. New York: McGraw-Hill.
74. Winfield, L. F. (1993) Investigating test content and curriculum content overlap to assess opportunity to learn. Journal of Negro Education, 62, 288–310.
75. Wolf, R. M. (1998) Validity issues in international assessments. International Journal of Educational Research, 29, 491–501.