Automatic generation of quantitative reasoning items: A pilot study

Journal of Individual Differences

Volumn 27, Issue 1, 2006, Pages 2-14

  Arendasy, Martin ^a   Sommer, Markus ^b   Gittler, Georg ^a   Hergovich, Andreas ^a  

^a UNIVERSITY OF VIENNA   (Austria)

^b SCHUHFRIED GMBH   (Austria)

Author keywords

Algebra word problems; Automatic item generation; Quantitative reasoning; Rasch model

Indexed keywords

EID: 36549070458     PISSN: 16140001     EISSN: None     Source Type: Journal    
DOI: 10.1027/1614-0001.27.1.2     Document Type: Article

References (85)

1. Ackerman P.L. Within-task intercorrelations of skill performance: Implications for predicting individual differences?. Journal of Applied Psychology 1989, 74:360-364.
2. Andersen E.B. A goodness of fit test for the Rasch model. Psychometrica 1973, 8:123-140.
3. Arbuckle J.L. Amos 5.0 update to the Amos user's guide 2003, Smallwaters Corporation, Chicago.
4. Arendasy M. TestWeb - Ein webbasiertes Testsystem [TestWeb - A web-based system for psychological assessment] 2002, Online, URL http://131.130.64.42/diffpsylabor/start.aspx (Feb. 23, 2003).
5. Arendasy M. AGen: Ein Itemgenerator zur Herstellung algebraischer Textaufgaben [AGen - An item generator for the construction of algebra word problems] 2004, Eigenverlag, Wien.
6. C. International Journal of Testing 2005, 5:197-224.
7. Arendasy, M. (in press). Automatisierte Itemgenerierung und psychometrische Qualitätssicherung am Beispiel des Matrizentests GEOM [Automatic Item generation and psychometric quality control demonstrated using the figural matrices test GEOM]. Frankfurt: Peter Lang Academic Publishers.
8. Arendasy M., Gittler G. IRT-basierter Vergleich zweier Varianten automationsgestützt erstellter Matrizentestaufgaben [IRT-based comparison of two automatically generated figural matrices variants]. Zeitschrift für Differentielle und Diagnostische Psychologie 2003, 24:261-274.
9. Arendasy M., Sommer M. The effect of different types of perceptual manipulations on the dimensionality of automatically generated figural matrices. Intelligence 2005, 33:307-324.
10. Arendasy M., Sommer M., Ponocny I. Psychometric approaches help resolve competing cognitive models: When less is more than it seems. Cognition & Instruction 2005, 23:503-521.
11. Baddeley A.D. Is working memory working?. Quarterly Journal of Experimental Psychology 1992, 44A(1):1-31.
12. Bejar I.I. Generative testing: From conception to implementation. Item generation for test development 2002, 199-218. Erlbaum, Hillsdale, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
13. Bernardo A.B.I. Problem-specific information and the development of problem-type schemata. Journal of Experimental Psychology: Learning Memory and Cognition 1994, 20:379-395.
14. Bickley P.G., Keith T.Z., Wolfle L.M. The three-stratum theory of cognitive abilities: Test of the structure of intelligence across the lifespan. Intelligence 1995, 20:309-328.
15. Blessing S.B., Ross B.H. Content effects in problem categorization and problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition 1996, 22:792-810.
16. Bobrow D.G. Natural language input for a computer problem-solving system. Semantic information processing 1968, 133-215. MIT Press, Cambridge, MA. M. Minsky (Ed.).
17. Browne M.W., Cudeck R. Alternative ways of assessing model fit. Testing structural equation models 1993, 445-455. Sage, Newbury Park, CA. K.A. Bollen, J.S. Long (Eds.).
18. Byrne B.M. Structural equation modeling with AMOS: Basic concepts, application, and programming 2001, Erlbaum, London.
19. Carroll J.B. Human cognitive abilities - A survey of factor analytic studies 1993, Cambridge University Press, Cambridge, UK.
20. Dennis I., Handley S., Bradon P., Evans J., Newstead S. Approaches to modeling item-generative tests. Item generation for test development 2002, 53-72. Erlbaum, Hillsdale, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
21. Edelblut, P., Elliot, S., Mikulas, C., & Bosley, L., (2002, June). Internet test security lessons learned. Paper presented at the International Conference on Computer-Based Testing and the Internet: Building Guidelines for Best Practice, Winchester, UK.
22. Embretson S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 1983, 93:179-197.
23. Embretson S.R., Reise S.P. Item response theory for psychologists 2000, Erlbaum, Mahwah, NJ.
24. Enright M.K., Morley M., Sheehan K.M. Item by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education 2002, 15(1):49-74.
25. Fischer G.H., Molenaar I.W. Rasch models. Foundations, recent developments, and applications 1995, Springer, New York.
26. Glas C.A.W., Verhelst N.D. Tests of fit for polytomous Rasch models. Rasch models, foundations, recent developments, and applications 1995, 325-352. Springer, New York. G.H. Fischer, I.W. Molenaar (Eds.).
27. Greeno J.G., Smith D.R., Moore J.L. Transfer of situated learning. Transfer on trial: Intelligence, cognition, and instruction 1993, 99-167. Ablex Publishing, Westpoint, CT. D.K. Dettermann, R.S. Sternberg (Eds.).
28. Gustafsson J.E. A unifying model of the structure of intellectual abilities. Intelligence 1984, 8:179-203.
29. Hall R., Kibler D., Wenger E., Truxaw C. Exploring the episodic structure of algebra story problem solving. Cognition and Instruction 1989, 6:223-283.
30. Harris, W.G., & Jackson, D.N., (2002). Security concerns for e-testing. Paper presented at the International Conference on Computer-Based Testing and the Internet: Building Guidelines for Best Practice, Winchester, UK.
31. Hinsley D.A., Hayes J.R., Simon H.A. From words to equations: Meaning and representation in algebra word problems. Cognitive processes in comprehension 1977, 89-105. Erlbaum, Hillsdale, NJ. M.A. Just, P.A. Carpenter (Eds.).
32. Holland P.W., Wainer H. Differential item functioning 1993, Erlbaum, Hillsdale, NJ.
33. Horn J.L. Models for intelligence. Intelligence: Measurement, theory, and public policy 1989, 29-73. University of Illinois Press, Urbana. R. Linn (Ed.).
34. Horn J.L., Noll J. Human cognitive capabilities: Gf-Gc theory. Contemporary intellectual assessment: Theory, tests, and issues 1997, 49-91. Guilford, New York. D.P. Flanagan, J.L. Genshaft, P.L. Harrison (Eds.).
35. Hornke L.F. Item-generative models for higher order cognitive functions. Item generation for test development 2002, 159-178. Erlbaum, Hillsdale, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
36. Hornke L.F., Arendasy M., Sommer M., Häusler J., Wagner-Menghin M., Gittler G., Bognar B., Wenzl M. Intelligenz-Struktur-Batterie (INSBAT): Eine Testbatterie zur Messung von Intelligenz [Intelligence Structure Battery (INSBAT): A test battery for the measurement of intelligence] 2004, Dr. Gernot Schuhfried GmbH, Mödling.
37. Hu L.T., Bentler P.M. Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 1999, 6:1-55.
38. Irvine S.H. The foundations of item generation for mass testing. Item generation for test development 2002, 3-34. Erlbaum, Mahwah, NJ. S.H. Irvine, P.C. Kyllonnen (Eds.).
39. Irvine S.H., Kyllonen P.C. Item generation for test development 2002, Erlbaum, Mahwah, NJ.
40. Katz I.R., Friedman D.E., Bennett R.E., Berger A.E. Differences in strategies used to solve stem-equivalent constructed response and multiple choice SAT-Mathematics items (College Board Report No. 95-3) 1996, College Entrance Board, New York.
41. Kintsch W. Comprehension: A paradigma for cognition 1998, Cambridge University Press, Cambridge, UK.
42. Kline R.B. Principles and practice of structural equation modeling 1998, Guilford, New York.
43. Koedinger K.R., Nathan M.J. The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences 2004, 13:129-164.
44. Lane S. Use of restricted item response models for examining item difficulty ordering and slope uniformity. Journal of Educational Measurement 1991, 28:295-309.
45. Lepik M. Algebraic word problems: Role of linguistic and structural variables. Educational Studies in Mathematics 1990, 21:83-90.
46. MacCallum R.C., Browne M.W., Sugawara H.M. Power analysis and determination of sample size for covariance structure modeling. Psychological Methods 1996, 1:130-149.
47. Marsh H.W., Hau K.T., Wen Z. In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralising Hu & Bentler's (1999) findings. Structural Equation Modelling 2004, 11:320-341.
48. Martin-Löf P. Statistiska Modeller. Anteckningar fran seminarier lasaret 1969-1970 utarbetade av Rolf Sundberg. Obetydligt ändrat nytryck, october 1973 [Statistical models: Report of the seminars 1969-1970 held by Rolf Sundberg, October 1973] 1973, Institutet för säkringsmatematik och matematisk statistik vid Stockholms universitet, Stockholm.
49. Mayer R.E. Frequency norms and structural analysis of algebra story problems into families, categories, and templates. Instructional Science 1981, 10:135-175.
50. Mayer R.E. Memory for algebra story problems. Journal of Educational Psychology 1982, 74:199-216.
51. Mayer R.E., Larkin J.H., Kadane J.B. A cognitive analysis of mathematical problem solving ability. Advances in the psychology of human intelligence 1984, 231-273. Erlbaum, Hillsdale, NJ. R.J. Sternberg (Ed.).
52. McGrew K.S., Woodcock R.W. Technical manual. Woodcock-Johnson III 2001, Riverside Publishing, Itasca, IL.
53. Müller-Philipp S., Tarnai C. Signifikanz und Relevanz von Modellabweichungen beim Rasch-Modell [Significance and relevance of model fit tests for the Rasch model]. Moderne Testtheorie [Modern test theory] 1989, 239-258. Psychologie Verlags Union, Weinheim. K.D. Kubinger (Ed.).
54. Nathan M.J., Kintsch W., Young E. A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction 1992, 9:329-389.
55. Nhouyvanisong, A., (1999). Enhancing mathematical competence and understanding: Using open ended problems and informal strategies. Doctoral dissertation, Psychology Department, Carnegie Mellon University.
56. Paige S.J., Simon H.A. Cognitive processes in solving algebra word problems. Problem solving 1966, 51-119. Wiley, New York. B. Kleinmutz (Ed.).
57. Phelps L., McGrew K., Knopik S., Ford L. The general (g), broad, and narrow CHC stratum characteristics of the WJ-III and WISC-III tests: A confirmatory cross-battery investigation. School Psychology Quarterly 2005, 20:58-66.
58. Ponocny I. Nonparametric goodness-of-fit tests for the Rasch model. Psychometrika 2001, 66:437-460.
59. Ponocny I., Ponocny-Seliger E. T-RASCH 1.0 1999, ProGAMMA, Groningen.
60. Rasch G. Probabilistic models for some intelligence and attainment tests 1980, The University of Chicago Press, Chicago, Original published 1960.
61. Reed S.K. A structure-mapping model for word problems. Journal of Experimental Psychology: Learning, Memory and Cognition 1987, 13:124-139.
62. Reusser, K., (1989). Vom Text zur Situation zur Gleichung. Kognitive Simulation von Sprachverständnis und Mathematisierung beim Lösen von Textaufgaben [Text, situation and equation: Cognition simulation of verbal comprehension for solving algebraic word problems]. Habilitationsschrift, Universität Bern, Switzerland.
63. Reusser K., Stebler R. Every word problem has a solution - the social reality of mathematical modelling in school. Learning and Instruction 1997, 28:309-328.
64. Roid G.H. Stanford-Binet intelligence scales 2003, Riverside, Itasca, IL. 5th ed.
65. Rost J. Lehrbuch Testtheorie Testkonstruktion. [Textbook of test theory and test construction] 2004, Huber, Bern. 2nd ed.
66. Sebrecht M.M., Enright M., Bennett E., Martin K. Using algebra word problems to assess quantitative ability: Attributes, strategies, and errors. Cognition and Instruction 1996, 14:285-343.
67. Schuler H. Personalauswahl im europäischen Vergleich. [Personnel selection: A European comparison]. Personalmanagement in Europa [Personell management in Europe] 2000, Hogrefe, Göttingen. E. Regnet, M.L. Hoffmann (Eds.).
68. Shalin V., Bee N.V. Analysis of the semantic structure of a domain of word problems (Tech. Rep. No. APS-20) 1985, University of Pittsburgh, Learning Research and Development Center, Pittsburgh.
69. Siegler R.S. Emerging minds: The process of change in children's thinking 1996, Oxford University Press, New York.
70. Spada H., Scheiblechner H. Stichprobenunabhängige Denkmodelle. Eine Analyse von Denkfehlern beim syllogistischen Schlussfolgern [Sample independent models of thinking: An analysis of errors in syllogistic reasoning]. Bericht über den 29. Kongress der deutschen Gesellschaft für Psychologie, Kiel 1970 [Report of the 29th Convention of the German Society of Psychology] 1973, Hogrefe, Göttingen. G. Reinert (Ed.).
71. Staub F.C., Reusser K. The role of presentational structure in understanding and solving mathematical word problems. Discourse comprehension. Essays in honor of Walter Kintsch 1995, 285-305. Erlbaum, Hillsdale, NJ. C.A. Weaver, S. Mannes, C.R. Fletcher (Eds.).
72. Srp, G., (1993). Syllogismen als Aufgaben für einen computerisierten adaptiven Test [Syllogism tasks for a computerized adaptive test]. Unpublished diploma, University of Vienna.
73. Stocking M.L., Lewis Ch. Methods of controlling the exposure of items in CAT. Computerized adaptive testing: Theory and practice 2000, Kluwer, Boston. W.J. Van der Linden, C.A.W. Glas (Eds.).
74. Sympson, J.B., & Hetter, R.D., (1985). Controlling item exposure rates in computerized adaptive testing. Paper presented at the annual meeting of the Military Testing Association. San Diego, CA: Navy Personnel Research and Development Center.
75. Tabachneck, H.J.M., Koedinger, K.R., & Nathan, M.J., (1994). Toward a theoretical account of strategy use and sense making in mathematics problem solving. In Proceedings of the 16th Annual Conference of the Cognitive Science Society (pp. 836- 841). Hillsdale, NJ: Erlbaum.
76. Tirre W.C., Pena C.M. Components of quantitative reasoning: General and group ability factors. Intelligence 1993, 17:501-521.
77. Tirre W.C., Field K.A. Structural models of abilities measured by the Ball Aptitude Battery. Educational and Psychological Measurement 2002, 62:830-856.
78. Undheim J.O., Gustafsson J.E. The hierarchical organisation of cognitive abilities: Restoring general intelligence through the use of linear structural relations (LISREL). Multivariate Behavioral Research 1987, 22:149-171.
79. van Dijk T.A., Kintsch W. Strategies of discourse comprehension 1983, Academic Press, New York.
80. Van der Linden W.J., Glas C.A.W. Computerized adaptive testing: Theory and practice 2000, Kluwer, Boston.
81. Van der Linden W.J., Hambleton R.K. Handbook of modern item response theory 1997, Springer, New York.
82. von Davier M. Methoden zur Prüfung probabilistischer Testmodelle [Model fit tests for probabilistic test models] 1996, IPN, Kiel.
83. Wagner-Menghin M. Manual Lexikon-Wissen Test (LEWITE). [Manual Lexical Knowledge Test (LEWITE] 2004, Dr. G. Schuhfried GmbH, Mödling.
84. Wainer H. Computerized adaptive testing: A primer 2000, Erlbaum, Hillsdale, NJ. 2nd ed.
85. Yerushalmy M., Gilead S. Structures of constant rate word problems: A functional approach analysis. Educational Studies in Mathematics 1999, 39(1-3):185-203.