Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school grades

Intelligence

Volumn 54, Issue , 2016, Pages 55-69

  Kretzschmar, André ^a   Neubert, Jonas C ^a,b   Wüstenberg, Sascha ^a   Greiff, Samuel ^a  

^a UNIVERSITY OF LUXEMBOURG   (Luxembourg)

^b BRANDENBURG UNIVERSITY OF TECHNOLOGY   (Germany)

Author keywords

Complex problem solving; Intelligence; Nested factor model; School grades; Validity

Indexed keywords

EID: 84949254831     PISSN: 01602896     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.intell.2015.11.004     Document Type: Article

References (90)

1. Amthauer R., Brocke B., Liepmann D., Beauducel A. I-S-T 2000 R - Intelligenz-Struktur-Test 2000 R [Intelligence Structure Test 2000 (revised)] 2001, Hogrefe, Göttingen.
2. Bahrick H.P., Hall L.K., Berger S.A. Accuracy and distortion in memory for high school grades. Psychological Science 1996, 7(5):265-271. http://doi.org/10.1111/j.1467-9280.1996.tb00372.x.
3. Beauducel A., Kersting M. Fluid and crystallized intelligence and the Berlin Model of Intelligence Structure (BIS). European Journal of Psychological Assessment 2002, 18(2):97-112. http://doi.org/10.1027//1015-5759.18.2.97.
4. Beauducel A., Brocke B., Liepmann D. Perspectives on fluid and crystallized intelligence: Facets for verbal, numerical, and figural intelligence. Personality and Individual Differences 2001, 30(6):977-994.
5. Beauducel A., Liepmann D., Felfe J., Nettelnstroth W. The impact of different measurement models for fluid and crystallized intelligence on the correlation with personality traits. European Journal of Psychological Assessment 2007, 23(2):71-78. http://doi.org/10.1027/1015-5759.23.2.71.
6. Beckmann J.F. Lernen und komplexes Problemlösen: Ein Beitrag zur Konstruktvalidierung von Lerntests. [Learning and complex problem solving. A contribution to the validation of learning tests] 1994, Holos, Bonn.
7. Beckmann J.F., Goode N. The benefit of being naïve and knowing it: The unfavourable impact of perceived context familiarity on learning in complex problem solving tasks. Instructional Science 2014, 42(2):271-290. http://doi.org/10.1007/s11251-013-9280-7.
8. Brunner M. No g in education?. Learning and Individual Differences 2008, 18(2):152-165. http://doi.org/10.1016/j.lindif.2007.08.005.
9. Brunner M., Süß H.-M. Analyzing the reliability of multidimensional measures: An example from intelligence research. Educational and Psychological Measurement 2005, 65(2):227-240. http://doi.org/10.1177/0013164404268669.
10. Brunner M., Nagy G., Wilhelm O. A tutorial on hierarchically structured constructs. Journal of Personality 2012, 80(4):796-846. http://doi.org/10.1111/j.1467-6494.2011.00749.x.
11. Buchner A., Funke J. Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology Section A 1993, 46(1):83-118. http://doi.org/10.1080/14640749308401068.
12. Campbell D.T., Fiske D.W. Convergent and discriminant validation by the multitrait multimethod matrix. Psychological Bulletin 1959, 56(2):81-105. http://doi.org/http://dx.doi.org/10.1037/h0046016.
13. Carroll J.B. Human cognitive abilities. A survey of factoranalytic studies 1993, Cambridge University Press, Cambridge.
14. Core Team R. R: A Language and Environment for Statistical Computing 2013, R Foundation for Statistical Computing, Vienna, Austria, Retrieved from http://www.R-project.org/.
15. Danner D., Hagemann D., Holt D.V., Hager M., Schankin A., Wüstenberg S., Funke J. Measuring performance in dynamic decision making. Journal of Individual Differences 2011, 32:225-233. http://doi.org/10.1027/1614-0001/a000055.
16. Danner D., Hagemann D., Schankin A., Hager M., Funke J. Beyond IQ: A latent state-trait analysis of general intelligence, dynamic decision making, and implicit learning. Intelligence 2011, 39:323-334. http://doi.org/10.1016/j.intell.2011.06.004.
17. Dörner D., Kreuzig H.W., Reither F., Stäudel T. Lohhausen: Vom Umgang mit Unbestimmtheit und Komplexität [Lohhausen: Dealing with uncertainty and complexity] 1983, Huber, Bern.
18. Eid M., Lischetzke T., Nussbeck F.W., Trierweiler L.I. Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods 2003, 8(1):38-60. http://doi.org/10.1037/1082-989X.8.1.38.
19. Fan X., Sivo S.A. Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research 2007, 42(3):509-529. http://doi.org/10.1080/00273170701382864.
20. Fischer A., Greiff S., Funke J. The process of solving complex problems. The Journal of Problem Solving 2012, 4(1):19-41. http://doi.org/10.7771/1932-6246.1118.
21. Fischer A., Greiff S., Wüstenberg S., Fleischer J., Buchwald F., Funke J. Assessing analytic and interactive aspects of problem solving competency. Learning and Individual Differences 2015, 39:172-179. http://doi.org/10.1016/j.lindif.2015.02.008.
22. Frensch P.A., Funke J. Definitions, traditions, and a general framework for understanding complex problem solving. Complex problem solving: The European perspective 1995, 3-25. Lawrence Erlbaum Associates, Hillsdale, NJ. P.A. Frensch, J. Funke (Eds.).
23. Funke J. Steuerung dynamischer Systeme durch Aufbau und Anwendung subjektiver Kausalmodelle [Control of dynamic systems by building up and using subjective causal models]. Zeitschrift für Psychologie 1985, 193:435-457.
24. Funke J. Dealing with dynamic systems: Research strategy, diagnostic approach and experimental results. The German Journal of Psychology 1992, 16(1):24-43.
25. Funke J. Wissen über dynamische Systeme: Erwerb, Repräsentation und Anwendung [Knowledge about dynamic systems: Acquisition, representation, and use] 1992, Springer, Berlin; New York.
26. Themenheft "komplexes problemlösen" [special issue: Complex problem solving]. Psychologische Rundschau 1999, 50(4).
27. Funke J. Dynamic systems as tools for analysing human judgement. Thinking and Reasoning 2001, 7(1):69-89. http://doi.org/10.1080/13546780042000046.
28. Funke J. Complex problem solving: A case for complex cognition?. Cognitive Processing 2010, 11:133-142. http://doi.org/10.1007/s10339-009-0345-0.
29. Gignac G.E. Raven's is not a pure measure of general intelligence: Implications for g factor theory and the brief measurement of g. Intelligence 2015, 52:71-79. http://doi.org/10.1016/j.intell.2015.07.006.
30. Gonzalez C., Thomas R.P., Vanyukov P. The relationships between cognitive ability and dynamic decision making. Intelligence 2005, 33(2):169-186. http://doi.org/10.1016/j.intell.2004.10.002.
31. Gottfredson L.S. Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography. Intelligence 1997, 24(1):13-23.
32. Greiff S. Individualdiagnostik der komplexen Problemlösefähigkeit [Assessment of problem solving ability] 2012, Waxmann, Münster.
33. Greiff S., Fischer A. Der Nutzen einer komplexen Problemlösekompetenz: Theoretische Überlegungen und empirische Befunde [The value of complex problem solving competency: Theoretical considerations and empirical results]. Zeitschrift für Pädagogische Psychologie 2013, 27(1):27-39. http://doi.org/10.1024/1010-0652/a000086.
34. Greiff S., Kretzschmar A., Müller J.C., Spinath B., Martin R. The computer-based assessment of complex problem solving and how it is influenced by students' information and communication technology literacy. Journal of Educational Psychology 2014, 106(3):666-680. http://doi.org/10.1037/a0035426.
35. Greiff S., Fischer A., Stadler M., Wüstenberg S. Assessing complex problem-solving skills with multiple complex systems. Thinking and Reasoning 2014, 21(3):356-382. http://doi.org/10.1080/13546783.2014.989263.
36. Greiff S., Fischer A., Wüstenberg S., Sonnleitner P., Brunner M., Martin R. A multitrait-multimethod study of assessment instruments for complex problem solving. Intelligence 2013, 41(5):579-596. http://doi.org/10.1016/j.intell.2013.07.012.
37. Greiff S., Stadler M., Sonnleitner P., Wolff C., Martin R. Sometimes less is more: Comparing the validity of complex problem solving measures. Intelligence 2015, 50:100-113. http://doi.org/10.1016/j.intell.2015.02.007.
38. Greiff S., Wüstenberg S., Funke J. Dynamic problem solving: a new assessment perspective. Applied Psychological Measurement 2012, 36(3):189-213. http://doi.org/10.1177/0146621612439620.
39. Greiff S., Wüstenberg S., Molnar G., Fischer A., Funke J., Csapo B. Complex problem solving in educational contexts-something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology 2013, 105(2):364-379. http://doi.org/10.1037/a0031856.
40. Guthke J. The learning test concept-an alternative to the traditional static intelligence test. German Journal of Psychology 1982, 6(4):306-324.
41. Heller K.A., Kratzmeier H., Lengfelder A. Matrizen-Test-Manual. Ein Handbuch zu den Standard Progressive Matrices von J.C. Raven [Matrices Test Manual. A Handbook of Standard Progressive Matrices of J.C. Raven] 1998, Beltz-Testgesellschaft, Göttingen.
42. Hossiep R., Schulte M. BOWIT. Bochumer Wissenstest [BOWIT. Bochumer Knowledge Test] 2008, Hogrefe, Göttingen.
43. Hu L., Bentler P.M. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 1999, 6(1):1-55. http://doi.org/10.1080/10705519909540118.
44. Jäger A.O. Intelligenzstrukturforschung: Konkurrierende Modelle, neue Entwicklungen, Perspektiven. [Structural research on intelligence: competing models, new developments, perspectives]. Psychologische Rundschau 1984, 35(1):21-35.
45. Jäger A.O., Süß H.-M., Beauducel A. Berliner Intelligenzstruktur-Test. Form 4 [Berlin Intelligence-Structure Test. Version 4] 1997, Hogrefe, Göttingen.
46. Jensen A.R., Wang L.-J. What is a good g?. Intelligence 1994, 18:231-258.
47. Kersting M. Zur Konstrukt- und Kriteriumsvalidität von Problemlöseszenarien anhand der Vorhersage von Vorgesetztenurteilen über die berufliche Bewährung [On the construct and criterion validity of problem-solving scenarios based on the prediction of supervisor assessment of job performance]. Diagnostica 2001, 47(2):67-76. http://doi.org/10.1026//0012-1924.47.2.67.
48. Kline P. An easy guide to factor analysis 1994, Routledge, London.
49. Kretzschmar A., Neubert J.C., Greiff S. Komplexes Problemlösen, schulfachliche Kompetenzen und ihre Relation zu Schulnoten [Complex problem solving, school competencies and their relation to school grades]. Zeitschrift für Pädagogische Psychologie 2014, 28(4):205-215. http://doi.org/10.1024/1010-0652/a000137.
50. Kröner S. Intelligenzdiagnostik per Computersimulation [Intelligence assessment via computer simulation] 2001, Waxmann, Münster.
51. Kröner S., Plass J.L., Leutner D. Intelligence assessment with computer simulations. Intelligence 2005, 33(4):347-368. http://doi.org/10.1016/j.intell.2005.03.002.
52. Kuncel N.R., Hezlett S.A., Ones D.S. Academic performance, career potential, creativity, and job performance: Can one construct predict them all?. Journal of Personality and Social Psychology 2004, 86(1):148-161. http://doi.org/10.1037/0022-3514.86.1.148.
53. Little T.D., Cunningham W.A., Shahar G., Widaman K.F. To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling 2002, 9(2):151-173. http://doi.org/10.1207/S15328007SEM0902_1.
54. Little T.D., Rhemtulla M., Gibson K., Schoemann A.M. Why the items versus parcels controversy needn't be one. Psychological Methods 2013, 18(3):285-300. http://doi.org/10.1037/a0033266.
55. Lu L., Weber H.S., Spinath F.M., Shi J. Predicting school achievement from cognitive and non-cognitive variables in a Chinese sample of elementary school children. Intelligence 2011, 39(2-3):130-140. http://doi.org/10.1016/j.intell.2011.02.002.
56. McGrew K.S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence 2009, 37(1):1-10. http://doi.org/10.1016/j.intell.2008.08.004.
57. Milling P.M. Modeling innovation processes for decision support and management simulation. System Dynamics Review 1996, 12(3):211-234.
58. Moshagen M., Musch J. Sample size requirements of the robust weighted least squares estimator. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 2014, 10(2):60-70. http://doi.org/10.1027/1614-2241/a000068.
59. Murray A.L., Johnson W. The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence 2013, 41(5):407-422. http://doi.org/10.1016/j.intell.2013.06.004.
60. Nachtigall C., Kroehne U., Funke F., Steyer R. Pros and cons of structural equation modeling. Methods of Psychological Research Online 2003, 8(2):1-22.
61. Neubert J.C., Kretzschmar A., Wüstenberg S., Greiff S. Extending the assessment of complex problem solving to finite state automata: Embracing heterogeneity. European Journal of Psychological Assessment 2015, 31(3):181-194. http://doi.org/10.1027/1015-5759/a000224.
62. Neubert J.C., Mainert J., Kretzschmar A., Greiff S. The assessment of 21st century skills in industrial and organizational psychology: complex and collaborative problem solving. Industrial and Organizational Psychology 2015, 18(2):238-268. 10.1017/iop.2015.14.
63. OECD Pisa 2012 results: Creative problem solving: Students' skills in tackling real-life problems (Volume V) 2014, OECD Publishing, Paris.
64. Pedhazur E.J. Multiple regression in behavioral research: Explanation and prediction 1997, Harcourt Brace College Publishers, Fort Worth.
65. Putz-Osterloh W. Über die Beziehung zwischen Testintelligenz und Problemlöseerfolg [On the relationship between test intelligence and success in problem solving]. Zeitschrift für Psychologie 1981, 189(1):79-100.
66. Raftery A.E. Bayesian model selection in social research. Sociological Methodology 1995, 25:111-164.
67. Raven J. Advanced progressive matrices 1958, Lewis, London. 2nd ed.
68. Reeve C.L., Blacksmith N. Identifying g: A review of current factor analytic practices in the science of mental abilities. Intelligence 2009, 37(5):487-494. http://doi.org/10.1016/j.intell.2009.06.002.
69. Reise S.P. The rediscovery of bifactor measurement models. Multivariate Behavioral Research 2012, 47(5):667-696. http://doi.org/10.1080/00273171.2012.715555.
70. Reise S.P., Bonifay W.E., Haviland M.G. Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment 2013, 95(2):129-140. http://doi.org/10.1080/00223891.2012.725437.
71. Revelle W. Psych: Procedures for Psychological, Psychometric, and Personality Research 2015, Northwestern University, Evanston, Illinois, (Retrieved from http://CRAN.R-project.org/package=psych).
72. Rosseel Y. lavaan: An R package for structural equation modeling. Journal of Statistical Software 2012, 48(2):1-36.
73. Schermelleh-Engel K., Moosbrugger H., Müller H. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online 2003, 8(2):23-74.
74. Shadish W.R., Cook T.D., Campbell D.T. Experimental and quasi-experimental designs for generalized causal inference 2001, Houghton Mifflin, Boston.
75. Sonnleitner P., Brunner M., Greiff S., Funke J., Keller U., Martin R., Latour T. The Genetics Lab: Acceptance and psychometric characteristics of a computer-based microworld assessing complex problem solving. Psychological Test and Assessment Modeling 2012, 54(1):54-72. http://doi.org/10.1037/e578442014-045.
76. Sonnleitner P., Keller U., Martin R., Brunner M. Students' complex problem-solving abilities: Their structure and relations to reasoning ability and educational success. Intelligence 2013, 41(5):289-305. http://doi.org/10.1016/j.intell.2013.05.002.
77. Soper D.S. Statistics Calculaters [Software] 2015, (Available from http://www.danielsoper.com/statcalc).
78. Sparfeldt J.R., Buch S.R., Rost D.H., Lehmann G. Akkuratesse selbstberichteter Zensuren [The accuracy of self-reported grades in school]. Psychologie in Erziehung und Unterricht 2008, 55:68-75.
79. Süß H.-M. Intelligenz, Wissen und Problemlösen: Kognitive Voraussetzungen für erfolgreiches Handeln bei computersimulierten Problemen [Intelligence, knowledge and problem solving: Cognitive prerequisites for successful behavior in computer-simulated problems] 1996, Hogrefe, Göttingen.
80. Süß H.-M. Intelligenz und komplexes Problemlösen: Perspektiven für eine Kooperation zwischen differentiell-psychometrischer und kognitionspsychologischer Forschung [Intelligence and complex problem solving: Perspectives for a cooperation between differential psychometric and cognition psychological research]. Psychologische Rundschau 1999, 50(4):220-228. http://doi.org/10.1026//0033-3042.50.4.220.
81. Süß H.-M. Prädiktive Validität der intelligenz im schulischen und außerschulischen Bereich [The predictive validity of intelligence in formal and non-formal education]. Perspektiven der Intelligenzforschung 2001, 109-135. Pabst, Lengerich. E. Stern, J. Guthke (Eds.).
82. Süß H.-M., Beauducel A. Intelligenztests und ihre Bezüge zu Intelligenztheorien. [Intelligence tests and their relationships to theories of intelligence]. Leistungs-, Intelligenz- und Verhaltensdiagnostik 2011, Vol. 3:97-234. Hogrefe, Göttingen. L.F. Hornke, M. Amelang, M. Kersting (Eds.).
83. Süß H.-M., Beauducel A. Modeling the construct validity of the Berlin Intelligence Structure Model. Estudos de Psicologia (Campinas) 2015, 32(1):13-25. http://doi.org/10.1590/0103-166X2015000100002.
84. Wallach D.P. Learning to control a coal-fired power plant: Empirical results and a model. Engineering psychology and cognitive ergonomics 1997, 2:82-90. Ashgate Publishers, Hampshire. D. Harris (Ed.).
85. Weiß R.H. CFT 20-R - Grundintelligenztest Skala 2 - Revision [Culture Fair Intelligence Test, Scale 2 (Cattell, R.B., 1960) - German revision] 2006, Hogrefe, Göttingen.
86. Wilhelm O. Measuring reasoning ability. Handbook of understanding and measuring intelligence 2005, 373-392. Sage, London. O. Wilhelm, R.W. Engle (Eds.).
87. Wirth J., Funke J. Dynamisches Problemlösen: Entwicklung und Evaluation eines neuen Messverfahrens zum Steuern komplexer Systeme [Dynamic problem solving: Developing and evaluating a new measurement device for controlling complex systems]. Problemlösekompetenz von Schülerinnen und Schülern: Diagnostische Ansätze, theoretische Grundlagen und empirische Befunde der deutschen PISA-2000-Studie 2005, 55-72. Verlag für Sozialwissenschaften, Wiesbaden. E. Klieme, D. Leutner, J. Wirth (Eds.).
88. Wittmann W.W., Hattrup K. The relationship between performance in dynamic systems and intelligence. Systems Research and Behavioral Science 2004, 21(4):393-409. http://doi.org/10.1002/sres.653.
89. Wittmann W.W., Süß H.-M. Investigating the paths between working memory, intelligence, knowledge, and complex problem-solving performances via Brunswik symmetry. Learning and individual differences: Process, trait and content determinants 1999, 77-104. APA, Washington. P.L. Ackerman, P.C. Kyllonen, R.D. Roberts (Eds.).
90. Wüstenberg S., Greiff S., Funke J. Complex problem solving - More than reasoning?. Intelligence 2012, 40(1):1-14. http://doi.org/10.1016/j.intell.2011.11.003.