Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items

Learning and Individual Differences

Volumn 17, Issue 4, 2007, Pages 366-383

  Arendasy, Martin ^a   Sommer, Markus ^a  

^a UNIVERSITY OF VIENNA   (Austria)

Author keywords

Algebra word problems; Automatic item generation; Educational assessment; Quantitative reasoning; Rasch Model

Indexed keywords

EID: 36549075025     PISSN: 10416080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.lindif.2007.03.005     Document Type: Article

References (80)

1. Ackerman P.L. Within-task intercorrelations of skill performance: Implications for predicting individual differences?. Journal of Applied Psychology 74 (1989) 360-364
2. Andersen E.B. A goodness of fit test for the Rasch Model. Psychometrica 8 (1973) 123-140
3. Arbuckle J.L. Amos 5.0 update to the Amos user's guide (2003), Smallwaters Corporation, Chicago, IL
4. Arendasy M. TestWeb - Ein webbasiertes Testsystem (2002). http://131.130.64.42/diffpsylabor/start.aspx [Online] Url: http://131.130.64.42/diffpsylabor/start.aspx (Feb. 23, 2003)
5. Arendasy M. Automatisierte Itemgenerierung und psychometrische Qualitätssicherung am Beispiel des Matrizentests GEOM (Automatic Item Generation and Psychometric Quality Control demonstrated using the figural matrices test GEOM) (2004), Habilitationsschrift der Universität Wien, Wien
6. C. International Journal of Testing 5 (2005) 197-224
7. Arendasy M., and Sommer M. The effect of different types of perceptual manipulations on the dimensionality of automatically generated figural matrices. Intelligence 33 3 (2005) 307-324
8. Arendasy, M., Sommer, M. (2007a, in revision). Two-component automatic item generation - An application to the field of quantitative reasoning.
9. Arendasy, M., Sommer, M. (2007b, in revision). Investigating the 'g' saturation of various stratum two factors using automated item generation.
10. Arendasy M., Sommer M., Gittler G., and Hergovich A. Automatic generation of quantitative reasoning items: Pilot study. Journal of Individual Differences 27 1 (2006) 2-14
11. Bejar I.I. Generative testing: From conception to implementation. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, New Jersey
12. Bernardo A.B.I. Problem-specific information and the development of problem-type schemata. Journal of Experimental Psychology. Learning, Memory, and Cognition 20 (1994) 379-395
13. Bickley P.G., Keith T.Z., and Wolfle L.M. The three-stratum theory of cognitive abilities: Test of the structure of intelligence across the life span. Intelligence 20 3 (1995) 309-328
14. Blessing S.B., and Ross B.H. Content effects in problem categorization and problem solving. Journal of Experimental Psychology. Learning, Memory, and Cognition 22 (1996) 792-810
15. Bobrow D.G. Natural language input for a computer problem-solving system. In: Minsky M. (Ed). Semantic information processing (1968), MIT Press, Cambridge 133-215
16. Browne M.W., and Cudeck R. Alternative ways of assessing model fit. In: Bollen K.A., and Long J.S. (Eds). Testing structural equation models (1993), Sage, Newbury Park 445-455
17. Byrne B.M. Structural equation modeling with AMOS: Basic concepts, application and programming (2001), Lawrence Erlbaum, London
18. Carroll J.B. Human cognitive abilities - A survey of factor analytic studies (1993), Cambridge University Press, Cambridge
19. Dennis I., Handley S., Bradon P., Evans J., and Newstead S. Approaches to modeling item-generative tests. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, New Jersey 53-72
20. Ebbinghaus H. Memory: A contribution to experimental psychology (1885), Teachers College Press, New York
21. Elman J.L., Bates E.A., Johnson M.H., Karmiloff-Smith A., Parisi D., and Plunkett K. Rethinking innateness: A connectionist perspective on development (1996), MIT Press, Cambridge
22. Embretson S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 93 (1983) 179-197
23. Embretson S.E. A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods 3 (1998) 380-396
24. Embretson S.E. Generating abstract reasoning items with cognitive theory. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, New Jersey 219-250
25. Embretson S.E., and Gorin J. Improving construct validity with cognitive psychology principles. Journal of Educational Measurement 38 (2001) 343-368
26. Embretson S.R., and Reise S.P. Item response theory for psychologists (2000), Lawrence Erlbaum Associates, Publishers, Mahwah, NJ
27. Enright M.K., Morley M., and Sheehan K.M. Item by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education 15 1 (2002) 49-74
28. Gittler G. Testpsychologische Aspekte der Raumvorstellungsforschung- Kritik, Lösungsansätze und empirische Befunde (Psychometric aspects of spatial ability research - critics, solutions and empirical results) (1992), Habilitationsschrift der Universität Wien, Wien
29. Glas C.A., and van der Linden W.J. Computerized adaptive testing with item cloning. Applied Psychological Measurement 27 (2003) 247-261
30. Graf E.A., Peterson S., Steffen M., and Lawless R. Psychometric and cognitive analysis as a basis for the design and revision of quantitative item models (ETS RR-05-25) (2005), ETS, Princeton
31. Greeno J.G., Smith D.R., and Moore J.L. In: D.K. (Ed). Transfer of situated learning (1993)
32. Gustafsson J.E. A unifying model of the structure of intellectual abilities. Intelligence 8 (1984) 179-203
33. Haenggi D. Differential aspects of visual short- and long-term memory. European Journal of Cognitive Psychology 1 (1989) 285-292
34. Hall R., Kibler D., Wenger E., and Truxaw C. Exploring the episodic structure of algebra story problem solving. Cognition and Instruction 6 (1989) 223-283
35. Hambelton R.K., and Swaminathan H. Item response theory: Principles and application (1985), Kluwer Academic Publishers, Boston
36. Holland P.W., and Wainer H. Differential item functioning (1993), Lawrence Erlbaum, Hillsdale, NJ
37. Horn J.L. Models for intelligence. In: Linn R. (Ed). Intelligence: Measurement, theory and public policy (1989), University of Illinois Press, Urbana 29-73
38. c Theory. In: Flanagan, D.P., Genshaft J.L., and Harrison P.L. (Eds). Contemporary intellectual assessment: theory, tests and issues (1997), The Guilford Press, New York 49-91
39. Hornke L.F. Item-generative models for higher order cognitive functions. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, New Jersey
40. Hornke L.F., Arendasy M., Sommer M., Häusler J., Wagner-Menghin M., Gittler G., et al. Intelligenz-Struktur-Batterie (INSBAT): Eine Testbatterie zur Messung von Intelligenz (2004), Dr. Gernot Schuhfried GmbH, Mödling
41. Hu L.T., and Bentler P.M. Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6 (1999) 1-55
42. Irvine S.H. The foundations of item generation for mass testing. In: Irvine S.H., and Kyllonnen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, Mahwah, NJ 3-34
43. Katz I.R., Friedman D.E., Bennett R.E., and Berger A.E. Differences in strategies used to solve stem-equivalent constructed response and multiple choice SAT-Mathematics items (College Board Report No. 95-3) (1996), College Entrance Board, New York
44. Kintsch W. Comprehension: A paradigma for cognition (1998), Cambridge University Press, Cambridge
45. Kline R.B. Principles and practice of structural equation modeling (1998), Guilford Press, New York
46. Koedinger K.R., and Nathan M.J. The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences 13 2 (2004) 129-164
47. Kosslyn S.M. The medium and the message in mental imagery: A theory. Psychological Review 81 (1981) 46-66
48. Lane S. Use of restricted item response models for examining item difficulty ordering and slope uniformity. Journal of Educational Measurement 28 (1991) 295-309
49. Lepik M. Algebraic word problems: Role of linguistic and structural variables. Educational Studies in Mathematics 21 1 (1990) 83-90
50. MacCallum, Browne, and Sugawara. Power analysis and determination of sample size for covariance structure modeling. Psychological Methods 1 (1996) 130-149
51. Marsh H.W., Hau K.T., and Wen Z. In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralising Hu and Bentler's (1999) findings. Structural Equation Modeling 11 (2004) 320-341
52. Martin-Löf, P. (1973). Statistiska Modeller. Anteckningar fran seminarier lasaret 1969-1970 utarbetade av Rolf Sundberg. Obetydligt ändrat nytryck, october 1973 (photocopied manuscript). Institutet för säkringsmatematik och matematisk statistik vid Stockholms universitet.
53. Mayer R.E. Frequency norms and structural analysis of algebra story problems into families, categories, and templates. Instructional Science 10 (1981) 135-175
54. Mayer R.E., Larkin J.H., and Kadane J.B. A cognitive analysis of mathematical problem solving ability. In: Sternberg R.J. (Ed). Advances in the psychology of human intelligence (1984), Lawrence Erlbaum, Hillsdale 231-273
55. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 50 (1995) 741-749
56. Müller-Philipp, S., Tarnai, C. (1989). Signifikanz und Relevanz von Modellabweichungen beim Rasch-Modell [Significance and relevance of deviations from the Rasch Model]. In K. D. Kubinger (Hrsg.), Moderne Testtheorie [Modern test theory](S. 239-258). Weinheim und München: Psychologie Verlagsunion.
57. Nathan M.J., and Koedinger K.R. Teachers' and researchers' beliefs about the development of Algebraic Reasoning. Journal of Research in Mathematics Education 31 (2000) 168-190
58. Nathan M.J., and Petrosino A. Expert blind spot among preservice teachers. American Educational Research Journal 40 (2003) 905-928
59. Nathan M.J., Kintsch W., and Young E. A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction 9 (1992) 329-389
60. Nhouyvanisong, A. (1999). Enhanching mathematical competence and understanding: Using open ended problems and informal strategies. Doctoral Dissertation. Psychology Department, Carnegie Mellon University.
61. Paige S.J., and Simon H.A. Cognitive processes in solving algebra word problems. In: Kleinmutz B. (Ed). Problem solving (1966), Wiley, New York 51-119
62. Paivio A. Imagery and verbal processes (1971), Holt, Rinehart, & Winston, New York
63. Phelps L., McGrew K., Knopik S., and Ford L. The general (g), broad, and narrow CHC stratum characteristics of the WJ-III and WISC-III tests: A confirmatory cross-battery investigation. School Psychology Quarterly 20 (2005) 58-66
64. Pellegrino J.W., Baxter G.P., and Glaser R. Addressing the "Two Disciplines" problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education 24 (1999) 307-353
65. Rasch G. Probabilistic models for some intelligence and attainment tests (1980), The University of Chicago Press, Chicago (Original published in 1960)
66. Reed S.K. A structure-mapping model for word problems. Journal of Experimental Psychology. Learning, Memory, and Cognition 13 (1987) 124-139
67. Reusser K. Vom Text zur Situation zur Gleichung. Kognitive Simulation von Sprachverständnis und Mathematisierung beim Lösen von Textaufgaben (From text to situation to equation: Cognitive simulations of text comprehension and mathematization in solving word problems) (1989), Habilitationsschrift, Universität Bern, Bern
68. Reusser K., and Stebler R. Every word problem has a solution - The social reality of mathematical modelling in school. Learning and Instruction 28 4 (1997) 309-328
69. Sebrecht M.M., Enright M., Bennett E., and Martin K. Using algebra word problems to assess quantitative ability: Attributes, strategies and errors. Cognition and Instruction 14 (1996) 285-343
70. Shalin V., and Bee N.V. Analysis of the semantic structure of a domain of word problems (Tech. Rep. No. APS-20) (1985), University of Pittsburgh, Learning Research and Development Center, Pittsburgh
71. Singley M.K., and Bennett T.E. Item generation and beyond: Application of schema theory to mathematics assessment. In: Irvine S.H., and Kyllonnen P.C. (Eds). Item generation for test development (2002), Lawrence Erlbaum Associates, Publishers, Mahwah, NJ 361-384
72. Sinharay S., and Johnson M. Analysis of data from an admissions test with item models (ETS RR-05-06) (2005), ETS, Princeton
73. Staub F.C., and Reusser K. The role of presentational structure in understanding and solving mathematical word problems. In: Weaver, C.A., Mannes S., and Fletcher C.R. (Eds). Discourse comprehension. Essays in honor of Walter Kintsch (1995), Lawrence Erlbaum, Hillsdale, NJ 285-305
74. Srp, G. (1993). Syllogismen als Aufgaben für einen computerisierten adaptiven Test. Unveröff. Dipl.Arbeit, Universität, Wien.
75. Tabachneck H.J.M., Koedinger K.R., and Nathan M.J. Toward a theoretical account of strategy use and sense making in mathematics problem solving. Proceedings of the 16th Annual Conference of the Cognitive Science Society (1994), Lawrence Erlbaum, Hillsdale 836-841
76. Tirre W.C., and Field K.A. Structural models of abilities measured by the Ball Aptitude Battery. Educational and Psychological Measurement 62 5 (2002) 830-856
77. Tirre W.C., and Pena C.M. Components of quantitative reasoning: General and group ability factors. Intelligence 17 (1993) 501-521
78. Undheim J.O., and Gustafsson J.E. The hierarchical organisation of cognitive abilities: Restoring general intelligence through the use of linear structural relations (LISREL). Multivariate Behavioral Research 22 (1987) 149-171
79. van der Linden W.J., and Hambleton R.K. Handbook of modern item response theory (1997), Springer, New York
80. Yerushalmy M., and Gilead S. Structures of constant rate word problems: A functional approach analysis. Educational Studies in Mathematics 39 1-3 (1999) 185-203