A multitrait-multimethod study of assessment instruments for complex problem solving

Intelligence

Volumn 41, Issue 5, 2013, Pages 579-596

  Greiff, Samuel ^a   Fischer, Andreas ^b   Wüstenberg, Sascha ^a   Sonnleitner, Philipp ^a   Brunner, Martin ^c   Martin, Romain ^a  

^a UNIVERSITY OF LUXEMBOURG   (Luxembourg)

^b UNIVERSITY OF HEIDELBERG   (Germany)

^c FREIE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Complex Problem Solving; Genetics Lab; MicroDYN; MicroFIN; Reasoning

Indexed keywords

EID: 84883180833     PISSN: 01602896     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.intell.2013.07.012     Document Type: Article

References (64)

1. Autor D.H., Levy F., Murnane R.J. The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics 2003, 118(4):1279-1333.
2. Baker E.L., O'Neil H.F. Measuring problem solving in computer environments: Current and future states. Computers in Human Behavior 2002, 18:609-622.
3. Beauducel A., Liepmann D., Horn S., Brocke B. Intelligence Structure Test 2010, Hogrefe, Oxford.
4. Buchner A. Basic topics and approaches to the study of complex problem solving. Complex problem solving: The European perspective 1995, 27-63. Erlbaum, Hillsdale, NJ. P.A. Frensch, J. Funke (Eds.).
5. Buchner A., Funke J. Finite state automata: Dynamic task environments in problem solving research. The Quarterly Journal of Experimental Psychology 1993, 46A:83-118.
6. Bunderson C.V., Inouye D.K., Olsen J.B. The four generations of computerized educational measurement. Educational Measurement 1989, 367-407. Macmillan Publishing, New York, NY. R.L. Linn (Ed.).
7. Campbell D.T., Fiske D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 1959, 54:297-312.
8. Carroll J.B. Human cognitive abilities. A survey of factoranalytic studies 1993, Cambridge University Press, Cambridge.
9. Danner D., Hagemann D., Schankin A., Hager M., Funke J. Beyond IQ. A latent state trait analysis of general intelligence, dynamic decision making, and implicit learning. Intelligence 2011, 39:323-334.
10. Dörner D. Diagnostik der operativen Intelligenz [Assessment of operative intelligence]. Diagnostica 1986, 32:290-308.
11. Dunbar K. Problem solving. A companion to cognitive science 1998, 289-298. Blackwell, London. W. Bechtel, G. Graham (Eds.).
12. Eid M., Lischetzke T., Nussbeck F.W. Structural equation models for multitrait-multimethod data. Handbook of multimethod measurement in psychology 2006, 283-299. APA, Washington, DC. M. Eid, E. Diener (Eds.).
13. Eid M., Lischetzke T., Nussbeck F.W., Trierweiler L.I. Separating trait effects from trait-specific method effects in multitrait-multimethod methods: A multiple-indicator CT-C(M-1) model. Psychological Methods 2003, 8(1):38-60.
14. Fischer A., Greiff S., Funke J. The process of solving complex problems. Journal of Problem Solving 2012, 4(1):19-41.
15. Funke J. Dynamic systems as tools for analysing human judgement. Thinking and Reasoning 2001, 7:69-89.
16. Funke J. Complex problem solving: A case for complex cognition?. Cognitive Processing 2010, 11:133-142.
17. Gottfredson L.S., Deary I.J. Intelligence predicts health and longevity, but why?. Current Directions in Psychological Science 2004, 13(1):1-4.
18. Greiff S., Fischer A. Der nutzen einer komplexen problemlösekompetenz: Theoretische überlegungen und empirische befunde [Usefulness of Complex Problem Solving competency: Theoretical considerations and empirical results]. Zeitschrift für Pädagogische Psychologie 2013, 27(1):1-13.
19. Greiff S., Holt D.V., Funke J. Perspectives on problem solving in cognitive research and educational assessment: analytical, interactive, and collaborative problem solving. Journal of Problem Solving 2013, 5:71-91.
20. Greiff S., Wüstenberg S., Funke J. Dynamic problem solving: A new measurement perspective. Applied Psychological Measurement 2012, 36:189-213.
21. Greiff S., Wüstenberg S., Molnar G., Fischer A., Funke J., Csapo B. Complex Problem Solving in educational settings - something beyond g: concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology 2013, 105:364-379.
22. Herrnstein R.J., Murray C. The bell curve. Intelligence and class structure in American life 1994, Free Press, New York.
23. Höfling V., Schermelleh-Engel K., Moosbrugger H. Analyzing multitrait-multimethod data. Methodology 2009, 5(3):99-111.
24. Hu L., Bentler P.M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 1999, 6:1-55.
25. Keller U., Sonnleitner P. Genetics Lab scoring algorithm 2012, University of Luxembourg, Luxembourg.
26. Kenny D.A., Kashy D.A. Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 1992, 112:165-172.
27. Klahr D. Exploring science: The cognition and development of discovery processes 2002, MIT press, Cambridge.
28. Klahr D., Dunbar K. Dual space search during scientific reasoning. Cognitive Science 1988, 12:1-48.
29. Klauer K., Phye G. Inductive reasoning: A training approach. Review of Educational Research 2008, 78(1):85-123.
30. Kline R.B. Principles and practice of structural equation modeling 2011, Guilford, New York, NY. 3rd ed.
31. Kröner S., Plass J.L., Leutner D. Intelligence assessment with computer simulations. Intelligence 2005, 33(4):347-368.
32. Kuhn D. Metacognitive development. Current Directions in Psychological Science 2000, 9(5):178-181.
33. Kuncel N., Hezlett S., Ones D. A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin 2001, 127:62-81.
34. Kyllonen P.C. New constructs, methods, and directions for computer-based assessment. The transition to computer-based assessment 2009, 151-156. Office for Official Publications of the European Communitites, Luxembourg. F. Scheuermann, J. Björnsson (Eds.).
35. Leighton J.P. Defining and describing reason. The Nature of Reasoning 2004, 3-11. Cambridge University Press, Cambridge. J.P. Leighton, R.J. Sternberg (Eds.).
36. Leutner D. The fuzzy relationship of intelligence and problem solving in computer simulations. Computers in Human Behavior 2002, 18:685-697.
37. Little T.D., Cunningham W.A., Shahar G., Widaman K.F. To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling 2002, 9(2):151-173.
38. Marsh H.W., Grayson D. Latent variable models of multitrait-multimethod data. Structural equation modeling 1995, 177-198. Sage, Thousand Oaks, CA. R.H. Hoyle (Ed.).
39. Mayer R.E., Wittrock M.C. Problem solving. Handbook of Educational Psychology 2006, 287-303. Lawrence Erlbaum, Mahwah, NJ. P.A. Alexander, P.H. Winne (Eds.).
40. McGrew K.S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence 2009, 37:1-10.
41. Müller H. Komplexes problemlösen: Reliabilität und Wissen [Complex problem solving: Reliability and knowledge] 1993, Holos, Bonn, Germany.
42. Muthén L.K., Muthén B.O. Mplus user's guide 2010, Muthén & Muthén, Los Angeles, CA. 6th ed.
43. Newell A., Simon H.A. Human problem solving 1972, Prentice-Hall, Englewood Cliffs, NJ.
44. Novick L.R., Bassok M. Problem solving. The Cambridge handbook of thinking and reasoning 2005, 321-349. University Press, Cambridge, NY. K.J. Holyoak, R.G. Morrison (Eds.).
45. OECD The Programme for International Student Assessment 2006 Accessed at:. http://www.oecd.org/dataoecd/15/13/39725224.pdf.
46. OECD PISA 2012 field trial problem solving framework Accessed at:. http://www.oecd.org/dataoecd/8/42/46962005.pdf.
47. Osman M. Controlling uncertainty: A review of human behavior in complex dynamic environments. Psychological Bulletin 2010, 136:65-86.
48. Putz-Osterloh W. Über die beziehung zwischen testintelligenz und problemlöseerfolg [On the relation between test intelligence and problem solving success]. Zeitschrift für Psychologie 1981, 189:79-100.
49. Raven J. Psychometrics, cognitive ability, and occupational performance. Review of Psychology 2000, 7:51-74.
50. Rigas G., Carling E., Brehmer B. Reliability and validity of performance measures in microworlds. Intelligence 2002, 30:463-480.
51. Roth P.L., Buster M.A., Bobko P. Updating the trainability tests literature on Black-White subgroup differences and reconsidering criterion-related validity. The Journal of Applied Psychology 2011, 96:34-45.
52. Sackett P.R. Personnel selection. Encyclopedia of psychology 2000, 152-156. Oxford University Press, New York, NY. A.E. Kazdin (Ed.).
53. Salas E., Cannon-Bowers J.A. The science of training: A decade of progress. Annual Review of Psychology 2001, 52:471-499.
54. Schmidt F.L., Hunter J. General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psychology 2004, 86(1):162-173.
55. Schweizer F., Wüstenberg S., Greiff S. Do Complex Problem Solving dimensions measure something over and above working memory capacity?. Learning and Individual Differences 2013, 24:42-52.
56. Sonnleitner P., Brunner M., Greiff S., Funke J., Keller U., Martin R., et al. The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving. Psychological Test and Assessment Modeling 2012, 54:54-72.
57. Sonnleitner P., Keller U., Martin R., Brunner M. Students' complex problem-solving abilities: Their structure and relations to reasoning ability and educational success. Intelligence 2013, 41:289-305.
58. Sternberg R.J., Berg C.A. Quantitative integration: Definitions of intelligence: A comparison of the 1921 and 1986 symposia. What is intelligence? Contemporary viewpoints on its nature and definition 1986, 155-162. Ablex, Norwood, New Jersey. R.J. Sternberg, D.K. Detterman (Eds.).
59. Williamson D.M., Bejar I.I., Mislevy R.J. Automated scoring of complex tasks in computer-based testing: An introduction 2006, Lawrence Erlbaum, Mahwah, NJ.
60. Wirth J., Klieme E. Computer-based assessment of problem solving competence. Assessment in Education: Principles, Policy & Practice 2003, 10(3):329-345.
61. Wittmann W., Hattrup K. The relationship between performance in dynamic systems and intelligence. Systems Research and Behavioral Science 2004, 21:393-440.
62. Wittmann W., Süß H.-M. Investigating the paths between working memory, intelligence, knowledge, and complex problem-solving performances via Brunswik symmetry. Learning and individual differences: Process, traits, and content determinants 1999, 77-108. APA, Washington, DC. P.L. Ackerman, P.C. Kyllonen, R.D. Roberts (Eds.).
63. Wüstenberg S., Greiff S., Funke J. Complex problem solving. More than reasoning?. Intelligence 2012, 40:1-14.
64. Yu C.-Y. Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes 2002, University of California, Los Angeles, CA.