Assessing complex problem-solving skills with multiple complex systems

Thinking and Reasoning

Volumn 21, Issue 3, 2015, Pages 356-382

  Greiff, Samuel ^a   Fischer, Andreas ^b   Stadler, Matthias ^a   Wüstenberg, Sascha ^a  

^a UNIVERSITY OF LUXEMBOURG   (Luxembourg)

^b UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

Complex problem solving; Knowledge acquisition; Knowledge application; MicroDYN; MicroFIN; Multiple complex systems

Indexed keywords

EID: 84929281106     PISSN: 13546783     EISSN: 14640708     Source Type: Journal    
DOI: 10.1080/13546783.2014.989263     Document Type: Article

References (98)

1. Autor, D. H., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118(4), 1279–1333.
2. Baddeley, A. (1989). The uses of working memory. In P. R. Solomon, G. R. Goethals, C. M. Kelley, & B. R. Stephens (Eds.), Memory: Interdisciplinary approaches (pp. 107–123). New York, NY: Springer.
3. Baumert, J., Evans, R. H., & Geiser, H. (1998). Technical problem solving among 10-year-old students as related to science achievement, out-of-school experience, domain-specific control beliefs, and attribution patterns. Journal of Research in Science Teaching, 35(9), 987–1013.
4. Beckmann, J. F. (1994). Lernen und komplexes Problemlösen [Learning and complex problem solving]. Bonn: Holos.
5. Berry, D. C., & Broadbent, D. E. (1984). On the relationship between task performance and associated verbalizable knowledge. TheQuarterly Journal of Experimental Psychology, 36A, 209–231.
6. Bornstein, M. H., Hahn, C. S., & Haynes, O. M. (2010). Social competence, externalizing, and internalizing behavioral adjustment from early childhood through early adolescence: Developmental cascades. Development and Psychopathology, 22(4), 717–735.
7. Broadbent, D., Fitzgerald, P., & Broadbent, M. H. P. (1986). Implicit and explicit knowledge in the control of complex systems. British Journal of Psychology, 77, 33–50.
8. Buchner, A. (1995). Basic topics and approaches to the study of complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 27–63). Hillsdale, NJ: Lawrence Erlbaum.
9. Buchner, A. (1999). Komplexes Problemlösen vor dem Hintergrund der Theorie finiter Automaten [Complex problem solving and the theory of finite state automata]. Psychologische Rundschau, 50, 206–212.
10. Buchner, A., & Funke, J. (1993). Finite state automata: Dynamic task environments in problem solving research. The Quarterly Journal of Experimental Psychology, 46A, 83–118.
11. Bühner, M. J., & Cheng, P. W. (2005). Causal learning. In K. J. Holyoak & R. G. Morrison (Eds.), The Cambridgehandbook ofthinking andreasoning (pp. 143–168). Cambridge: Cambridge University Press.
12. Bühner, M., Kröner, S., & Ziegler, M. (2008). Working memory, visual–spatial intelligence and their relationship to problem-solving. Intelligence, 36(4), 672–680.
13. Bühner, M., Krumm, S., & Pick, M. (2005). Reasoning = working memory≠ attention. Intelligence, 33(3), 251–272.
14. Carmines, E. G., & Zeller, R. A. (1991). Reliability and viability assessment. Thousand Oaks, CA: Sage.
15. Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70(6), 426.
16. Daneman, M., & Merikle, P. M. (1996). Working memory and language comprehension: A meta-analysis. Psychonomic Bulletin & Review, 3(4), 422–433.
17. Daniel, R. C., & Embretson, S. E. (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34(5), 348–364.
18. Danner, D., Hagemann, D., Schankin, A., Hager, M., & Funke, J. (2011). Beyond IQ: A latent state-trait analysis of general intelligence, dynamic decision making, and implicit learning. Intelligence, 39(5), 323–334.
19. Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571.
20. Dörner, D. (1986). Diagnostik der operativen Intelligenz [Assessment of operative intelligence]. Diagnostica, 32(4), 290–308.
21. Dörner, D. (1989). Die Logik des Misslingens. Strategisches Denken in komplexen Situationen [Logic of failure. Strategic thinking in complex situations]. Hamburg: Rowohlt.
22. Dörner, D., Kreuzig, H. W., Reither, F., & Stäudel, T. (1983). Lohhausen: Vom Umgang mit Komplexität [Lohhausen: On handling complexity]. Bern: Huber.
23. Dunbar, K., & Fugelsang, J. (2005). Scientific thinking and reasoning. In K. L. Holyoak & R. G. Morrison (Eds.), The Cambridgehandbook ofthinking andreasoning (pp. 705–725). New York, NY: Cambridge University Press.
24. Duncker, K. (1945). On problem solving. Psychological Monographs, 58(5), i–113.
25. Edwards, W. (1962). Dynamic decision theory and probabilistic information processing. Human Factors, 4, 59–73.
26. Even, R. (1993). Subject-matter knowledge and pedagogical content knowledge: Prospective secondary teachers and the function concept. Journal of Research in Mathematics Education, 24(2), 94–116.
27. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4(1), 19–42.
28. Frensch, P. A., & Funke, J. (Eds.). (1995). Complex problem solving. The European perspective. Hillsdale, NJ: Lawrence Erlbaum.
29. Fry, A. F., & Hale, S. (1996). Processing speed, working memory, and fluid intelligence: Evidence for a developmental cascade. Psychological Science, 7(4), 237–241.
30. Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking&Reasoning, 7(1), 69–89.
31. Funke, J. (2003). Problemlösendes Denken [Problem solving thinking]. Stuttgart: Kohlhammer.
32. Funke, J. (2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11(2), 133–142.
33. Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1, 107–143.
34. Gonzalez, C., Lerch, F. J., & Lebiere, C. (2003). Instance-based learning in dynamic decision making. Cognitive Science, 27, 591–635.
35. Greiff, S., & Fischer, A. (2013). Der Nutzen einer Komplexen Problemlösekompetenz: Theoretische Überlegungen und empirische Befunde [Usefulness of complex problem solving competency: Theoretical considerations and empirical results]. Zeitschrift für Pädagogische Psychologie, 27(1), 27–39.
36. Greiff, S., Fischer, A., Wüstenberg, S., Sonnleitner, P., Brunner, M., & Martin, R. (2013). A multitrait–multimethod study of assessment instruments for complex problem solving. Intelligence, 41, 579–596.
37. Greiff, S., & Funke, J. (2010). Systematische Erforschung komplexer Problemlösefähigkeit anhand minimal komplexer Systeme [Systematic research on complex problem solving by means of minimal complex systems]. Zeitschrift für Pädagogik, 56, 216–227.
38. Greiff, S., Krkovic, K., & Nagy, G. (2014). The systematic variation of task characteristics facilitates the understanding of task difficulty: A cognitive diagnostic modeling approach to complex problem solving. Psychological Test and Assessment Modeling, 56(1), 83–103.
39. st century skills. Heidelberg: Springer.
40. Greiff, S., Wüstenberg, S., & Funke, J. (2011, April). Measuring a cross-curricular skill:Competence models and construct validity of dynamic problem solving. Paper presented at the Conference of the American Educational Research Association in New Orleans, USA.
41. Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new measurement perspective. Applied Psychological Measurement, 36(3), 189–213.
42. Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapo, B. (2013). Complex problem solving in educational settings – something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105(2), 364–379.
43. Holyoak, K. J. (1985). The pragmatics of analogical transfer. In G. H. Bower (Ed.), The psychology of learning and motivation (pp. 59–87). New York, NY: Academic Press.
44. Horn, J. L., & Cattell, R. B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5), 253–270.
45. Jäger, A. O. (1973). Dimensionen der Intelligenz. Göttingen: Hogrefe Verlag für Psychologie.
46. Jeffries, R. P., Polson, P. G., Razran, L., & Atwood, M. E. (1977). A process model for missionaries: Cannibals and other river-crossing problems. Cognitive Psychology, 9, 412–440.
47. Kahneman, D. (2011). Thinking fast and slow. New York, NY: Farrar, Straus and Giroux.
48. Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1–48.
49. Klauer, K. C. (1993). Belastung und Entlastung beim Problemlösen. Eine Theorie des deklarativen Vereinfachens [Charge and discharge in problem solving. A theory of declarative simplification]. Göttingen: Hogrefe.
50. Kluge, A. (2008). Performance assessment with microworlds and their difficulty. Applied Psychological Measurement, 32, 156–180.
51. Kröner, S., Plass, J. L., & Leutner, D. (2005). Intelligence assessment with computer simulations. Intelligence, 33(4), 347–368.
52. Kuhn, D. (2000). Metacognitive development. CurrentDirections in Psychological Science, 9(5), 178–181.
53. Kyllonen, P. C., & Christal, R. E. (1990). Reasoning ability is (little more than) working-memory capacity?! Intelligence, 14(4), 389–433.
54. Leighton, J. P., & Sternberg, R. J. (Eds.). (2004). The nature of reasoning. Cambridge: Cambridge University Press.
55. MacGregor, J. N., Ormerod, T. C., & Chronicle, E. P. (2001). Information-processing and insight: A process model of performance on the nine-dot and related problems. Journal of Experimental Psychology: Learning, Memory and Cognition, 27, 176–201.
56. Marshalek, B., Lohman, D. F., & Snow, R. E. (1983). The complexity continuum in the radex and hierarchical models of intelligence. Intelligence, 7(2), 107–127.
57. Mayer, R. E., Larkin, J., & Kadane, J.B. (1984). A cognitive analysis of mathematical problem solving ability. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (Vol. 2, pp. 231–273). Hillsdale, NJ: Lawrence Erlbaum.
58. Mayer, R. E., & Wittrock, M. C. (2006). Problem solving. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 287–303). Hillsdale, NJ: Lawrence Erlbaum.
59. McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37(1), 1–10.
60. Neubert, J. C., Kretzschmar, A., Wüstenberg, S., & Greiff, S. (2014). Extending the assessment of complex problem solving to finite state automata: Embracing heterogeneity. European Journal of Psychological Assessment. Advanced online publication. doi: 10.1027/1015-5759/a000224
61. Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.
62. Novick, L. R., & Bassok, M. (2005). Problem solving. In K. J. Holyoak & R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 321–349). Cambridge: Cambridge University Press.
63. Novick, L. R., Hurley, S. M., & Francis, M. (1999). Evidence for abstract, schematic knowledge of three spatial diagram representation. Memory and Cognition, 27, 288–308.
64. Organisation for Economic Co-operation and Development. (2010). PISA 2012 problem solving framework (draft for field trial). Paris: Author.
65. Ohlsson, S. (1992). Information processing explanations of insight and related phenomena. In M. T. Keane & K. J. Gilhooly (Eds.), Advances in the psychology of thinking. London: Harvester Wheatsheaf.
66. Ohlsson, S. (2012). The problems with problem solving: Reflections on the rise, current status, and possible future of a cognitive research paradigm. Journal of Problem Solving, 5(1), 101–128.
67. Osman, M. (2010). Controlling uncertainty: A review of human behavior in complex dynamic environments. Psychological Bulletin, 136, 65–86.
68. Papert, S. (1980). Computer-based microworlds as incubators for powerful ideas. In R. Taylor (Ed.), The computer in the school: Tutor, tool, tutee (pp. 203–210). New York, NY: Teacher's College Press.
69. Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context-bound? Educational Researcher, 18(10), 16–25.
70. Quesada, J., Kintsch, W., & Gomez, E. (2005). Complex problem solving: A field in search of a definition? Theoretical Issues in Ergonomic Science, 6(1), 5–33.
71. Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago, IL: University of Chicago Press.
72. Raven, J. (2000). Psychometrics, cognitive ability, and occupational performance. Review of Psychology, 7, 51–74.
73. Rigas, G., & Brehmer, B. (1999). Mental processes in intelligence tests and dynamic decision making tasks. In P. Juslin & H. Montgomery (Eds.), Judgement and decision making: Neo-Brunswikian and process-tracing approaches (pp. 45–65). Hillsdale, NJ: Lawrence Erlbaum
74. Scherer, R., & Tiemann, R. (2012). Factors of problem-solving competency in a virtual chemistry environment: The role of metacognitive knowledge about strategies. Computers & Education, 59(4), 1199–1214.
75. Schunn, C. D., & Klahr, D. (1995). A 4-space model of scientific discovery. In Proceedings of the 17th Annual Conference of the Cognitive Science Society (pp. 106–111). Hillsdale, NJ: Erlbaum.
76. Schunn, C. D., & Klahr, D. (2000). Discovery processes in a more complex task. In D. Klahr (Ed.), Exploring science: The cognition and development of discovery processes (pp. 161–199). Cambridge, MA: MIT Press.
77. Schweizer, F., Wüstenberg, S., & Greiff, S. (2013). Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity. Learning & Individual Differences, 24, 42–52.
78. Simon, H. A. (1975). The functional equivalence of problem solving skills. Cognitive Psychology, 7(2), 268–288.
79. Snow, R. E. (1989). Toward assessment of cognitive and conative structures in learning. Educational Researcher, 18(9), 8–14.
80. Sonnleitner, P., Brunner, M., Greiff, S., Funke, J., Keller, U., Martin, R., Hazotte, C., Mayer, H., & Latour, T. (2012). The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving. Psychological Test and Assessment Modeling, 54(1), 54–72.
81. Sternberg, R. J. (1995). Expertise in complex problem solving: A comparison of alternative concepts. In P. A. Frensch & J. Funke (Eds.), Complex problem solving. The European perspective (pp. 295–321). Hillsdale, NJ: Lawrence Erlbaum.
82. Sternberg, R. J., & Berg, C. A. (Eds.). (1992). Intellectual development. Cambridge: Cambridge University Press.
83. Sugrue, B. (1995). A theory-based framework for assessing domain-specific problem-solving ability. Educational Measurement Issues and Practice, 14(3), 29–36.
84. Süß, H.-M. (1996). Intelligenz, Wissen und Problemlösen. Kognitive Voraussetzungen für erfolgreiches Handeln bei computersimulierten Problemen [Intelligence, knowledge, and problem solving: Cognitive prerequisites for success in problem solving with computer-simulated problems]. Göttingen: Hogrefe.
85. Süß, H. M. (1999). Intelligenz und komplexes problemlösen [Intelligence and complex problem solving]. Psychologische Rundschau, 50(4), 220–228.
86. Süß, H.-M. (2001). Prädikative Validität der Intelligenz im schulischen und außerschulischen Bereich. In E. Stern & J. Guthke (Eds.), Perspektiven der Intelligenzforschung [Perspectives on intelligence research] (pp. 109–135). Lengerich: Pabst Science.
87. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.
88. Tschirgi, J. E. (1980). Sensible reasoning: A hypothesis about hypotheses. Child Development, 51, 1–10.
89. Wagener, D. (2001). Psychologischediagnostik mit komplexenszenarios [Psychological diagnostic using complex scenarios]. Lengerich: Pabst.
90. Wason, P. C., & Shapiro, D. (1971). Natural and contrived experience in a reasoning problem. The Quarterly Journal of Experimental Psychology, 23(1), 63–71.
91. Weaver, W. (1948). Science and complexity. American Scientist, 36, 536–544.
92. Wenke, D., Frensch, P. A., & Funke, J. (2005). Complex problem solving and intelligence: Empirical relation and causal direction. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition and intelligence: Identifying the mechanisms of the mind (pp. 160–187). New York, NY: Cambridge University Press.
93. Wirth, J. (2004). Selbstregulation von Lernprozessen [Self-regulation of learning processes]. Münster: Waxmann.
94. Wirth, J., & Klieme, E. (2003). Computer-based assessment of problem solving competence. Assessment in Education: Principles, Policy, & Practice, 10, 329–345.
95. Wittmann, W. W., & Hattrup, K. (2004). The relationship between performance in dynamic systems and intelligence. Systems Research and Behavioral Science, 21(4), 393–409.
96. Wittmann, W., & Süß, H.-M. (1999). Investigating the paths between working memory, intelligence, knowledge, and complex problem-solving performances via Brunswik symmetry. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual differences: Process, traits, and content determinants (pp. 77–108). Washington, DC: APA.
97. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving -- more than reasoning? Intelligence, 40, 1–14.
98. Wüstenberg, S., Stadler, M., Hautamäki, J., & Greiff, S. (2014). The role of strategy knowledge for the application of strategies in complex problem solving tasks. Technology, Knowledge and Learning, 19, 127–146.