The computer-based assessment of complex problem solving and how it is influenced by students' information and communication technology literacy

Journal of Educational Psychology

Volumn 106, Issue 3, 2014, Pages 666-680

  Greiff, Samuel ^a   Kretzschmar, André ^a   Müller, Jonas C ^a   Spinath, Birgit ^b   Martin, Romain ^a  

^a UNIVERSITY OF LUXEMBOURG   (Luxembourg)

^b UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

Complex problem solving; ICT literacy; MicroDYN; Reasoning; Working memory

Indexed keywords

EID: 84894277822     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0035426     Document Type: Article

References (73)

1. Appel, M. (2012). Are heavy users of computer games and social media more computer literate? Computers & Education, 59, 1339-1349. doi: 10.1016/j.compedu.2012.06.004
2. Autor, D., & Dorn, D. (2009). This job is 'getting old': Measuring changes in job opportunities using occupational age structure. American Economic Review, 99, 45-51. doi:10.1257/aer.99.2.45
3. Autor, D., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change: An empirical investigation. The Quarterly Journal of Economics, 118, 1279-1333. doi:10.1162/003355303322552801
4. Ballantine, J. A., McCourt Larres, P., & Oyelere, P. (2007). Computer usage and the validity of self-assessed computer competence among first-year business students. Computers & Education, 49, 976-990. doi:10.1016/j.compedu.2005.12.001
5. Beauducel, A., Liepmann, D., Horn, S., & Brocke, B. (2010). Intelligence Structure Test. Oxford, England: Hogrefe.
6. Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: Wiley.
7. Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IA: American College Testing.
8. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18, 32-42. doi:10.3102/ 0013189X018001032
9. Buchner, A., & Funke, J. (1993). Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology Section A, 46, 83-118. doi:10.1080/ 14640749308401068
10. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge, England: Cambridge University Press. doi:10.1017/ CBO9780511571312
11. Clariana, R., & Wallace, P. (2002). Paper-based versus computer-based assessment: Key factors associated with the test mode effect. British Journal of Educational Technology, 33, 593-602. doi:10.1111/1467- 8535.00294
12. Danner, D., Hagemann, D., Schankin, A., Hager, M., & Funke, J. (2011). Beyond IQ: A latent state-trait analysis of general intelligence, dynamic decision making, and implicit learning. Intelligence, 39, 323-334. doi: 10.1016/j.intell.2011.06.004
13. Dörner, D., Kreuzig, H. W., Reither, F., & Stäudel, T. (1983). Lohhausen: Vom Umgang mit Unbestimmtheit und Komplexität [Lohhausen. On dealing with uncertainty and complexity]. Stuttgart, Germany: Huber.
14. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4, 19-41.
15. Frensch, P. A., & Funke, J. (1995). Complex problem solving: The European perspective. Hillsdale, NJ: Lawrence Erlbaum Associates.
16. Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7, 69-89. doi:10.1080/13546780042000046
17. Funke, J. (2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11, 133-142. doi:10.1007/s10339-009-0345-0
18. Goldhammer, F., Naumann, J., & Keßel, Y. (2013). Assessing individual differences in basic computer skills. European Journal of Psychological Assessment, 29, 236-275. doi:10.1027/1015-5759/a000153
19. Gonzalez, C., Thomas, R. P., & Vanyukov, P. (2005). The relationships between cognitive ability and dynamic decision making. Intelligence, 33, 169-186. doi:10.1016/j.intell.2004.10.002
20. Greiff, S., Fischer, A., Wüstenberg, S., Sonnleitner, P., Brunner, M., & Martin, R. (2013). A multitrait-multimethod study of assessment instruments for complex problem solving. Intelligence, 41, 579-596. doi: 10.1016/j.intell.2013.07.012
21. Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new assessment perspective. Applied Psychological Measurement, 36, 189-213. doi:10.1177/0146621612439620
22. Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapó, B. (2013). Complex problem solving in educational contexts-Something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105, 364 -379. doi: 10.1037/a0031856
23. Hartig, J., & Klieme, E. (2005). Die Bedeutung schulischer Bildung und soziobiographischer Merkmale für die Problemlösekompetenz [The importance of school education and sociobiographical characteristics for problem solving skills]. In E. Klieme, D. Leutner, & J. Wirth (Eds.), Problemlösekompetenz von Schülerinnen und Schülern. Diagnostische Ansätze, theoretische Grundlagen und empirische Befunde der deutschen PISA-2000-Studie (pp. 83-97). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
24. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118
25. Kersting, M. (1998). Differentielle Aspekte der sozialen Akzeptanz von Intelligenztests und Problemlöseszenarien als Personalauswahlverfahren [Differential-psychological aspects of applicants' acceptance of intelligence tests and problem solving scenarios as diagnostic tools for personnel selection]. Zeitschrift für Arbeits- und Organisationspsychologie, 42, 61-75.
26. Kröner, S., Plass, J., & Leutner, D. (2005). Intelligence assessment with computer simulations. Intelligence, 33, 347-368. doi:10.1016/j.intell .2005.03.002
27. Kuhn, D. (2009). Do students need to be taught how to reason? Educational Research Review, 4, 1-6. doi:10.1016/j.edurev.2008.11.001
28. Kyllonen, P. C. (2009). New constructs, methods, and directions for computer-based assessment. In F. Scheuermann & J. Björnsson (Eds.), The transition to computer-based assessment (pp. 151-156). Luxembourg, Luxembourg: Office for Official Publications of the European Communitites.
29. Leutner, D. (2002). The fuzzy relationship of intelligence and problem solving in computer simulations. Computers in Human Behavior, 18, 685-697. doi:10.1016/S0747-5632(02)00024-9
30. Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151-173. doi:10.1207/ S15328007SEM0902_1
31. Mayer, R. E., & Wittrock, M. C. (2006). Problem solving. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 287-303). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
32. McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1-10. doi:10.1016/j.intell.2008.08.004
33. Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114, 449-458. doi:10.1037/0033-2909.114.3.449
34. Michael, D. R., & Chen, S. L. (2006). Serious games: Games that educate, train, and inform. Boston, MA: Thomson Course Technology.
35. Molnár, G., Greiff, S., & Csapó, B. (2013). Inductive reasoning, domain specific and complex problem solving: Relations and development. Thinking Skills and Creativity, 9, 35-45. doi:10.1016/j.tsc.2013.03.002
36. Morris, L. W., Davis, M. A., & Hutchings, C. H. (1981). Cognitive and emotional components of anxiety: Literature review and a revised worry-emotionality scale. Journal of Educational Psychology, 73, 541-555. doi:10.1037/0022-0663.73.4.541
37. Muthén, L. K., & Muthén, B. O. (2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
38. Naumann, J., Richter, T., & Groeben, N. (2001). Validierung des INCOBI anhand eines Vergleichs von Anwendungsexperten und Anwendungsnovizen [Validation of the INCOBI trough comparison of expert and novice computer users]. Zeitschrift für Pädagogische Psychologie, 15, 219-232. doi:10.1024//1010-0652.15.34.219
39. Novick, L. R., & Bassok, M. (2005). Problem solving. In K. J. Holyoak & R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 321-349). Cambridge, England: Cambridge University Press.
40. Oberauer, K., Süß, H.-M., Schulze, R., Wilhelm, O., & Wittmann, W. W. (2000). Working memory capacity-facets of a cognitive ability construct. Personality and Individual Differences, 29, 1017-1045. doi: 10.1016/S0191-8869(99)00251-2
41. Oberauer, K., Süß, H.-M., Wilhelm, O., & Wittmann, W. W. (2003). The multiple faces of working memory: Storage, processing, supervision, and coordination. Intelligence, 31, 167-193. doi:10.1016/S0160- 2896(02)00115-0
42. Oberauer, K., Süß, H.-M., Wilhelm, O., & Wittmann, W. W. (2008). Which working memory functions predict intelligence? Intelligence, 36, 641-652. doi:10.1016/j.intell.2008.01.007
43. Organisation for Economic Co-Operation and Development. (2007). PISA 2006 science competencies for tomorrow's world. Paris, France: Author.
44. Organisation for Economic Co-Operation and Development. (2009a). Chapter 1: Programme for International Student Assessment: An overivew. Retrieved from http://www.oecd.org/berlin/42174841.pdf
45. Organisation for Economic Co-Operation and Development. (2009b). PIAAC problem solving in technology-rich environments. Retrieved from http:// search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage= en&cote=edu/wkp%282009%2915
46. Organisation for Economic Co-Operation and Development. (2010). PISA 2012 field trial problem solving framework. Retrieved from http://www .oecd.org/pisa/pisaproducts/46962005.pdf
47. Organisation for Economic Co-Operation and Development. (2012). Draft PISA 2015 collaborative problem solving assessment framework. Retrieved from http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf
48. Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2002). Practical considerations in computer-based testing. New York, NY: Springer.
49. Prensky, M. (2001). Digital natives, digital immigrants. Part 1. On the Horizon, 9, 1-6. doi:10.1108/10748120110424816
50. Raven, J. (1958). Advanced progressive matrices (2nd ed.). London, England: Lewis.
51. Raven, J. (2000). Psychometrics, cognitive ability, and occupational performance. Review of Psychology, 7, 51-74.
52. Raven, J., Raven, J. C., & Court, J. H. (1998). Manual for Raven's progressive matrices and vocabulary scales: Section 4. The advanced progressive matrices. San Antonio, TX: Harcourt Assessment.
53. Richter, T., Naumann, J., & Groeben, N. (2001). Das Inventar zur Computerbildung (INCOBI): Ein Instrument zur Erfassung von Computer Literacy und computerbezogenen Einstellungen bei Studierenden der Geistes- und Sozialwissenschaften [The Computer Literacy Inventory - An instrument for the assessment of computer literacy and attitudes toward the computer in students of the humanities and social sciences]. Psychologie in Erziehung und Unterricht, 48, 1-13.
54. Richter, T., Naumann, J., & Horz, H. (2010). Eine revidierte Fassung des Inventars zur Computerbildung (INCOBI-R) [A revised version of the Computer Literacy Inventory]. Zeitschrift für Pädagogische Psychologie, 24, 23-37. doi:10.1024/1010-0652/a000002
55. Russell, M., Goldberg, A., & O'Connor, K. (2003). Computer-based testing and validity: A look back into the future. Assessment in Education: Principles, Policy & Practice, 10, 279 -293. doi:10.1080/ 0969594032000148145
56. Sander, N. (2005). Inhibitory and executive functions in cognitive psychology: An individual differences approach examining structure and overlap with working memory capacity and intelligence. Aachen, Germany: Shaker.
57. Scheuermann, F., & Björnsson, J. (2009). The transition to computer-based assessment. Luxembourg, Luxembourg: Office for Official Publications of the European Communitites.
58. Schweizer, F., Wüstenberg, S., & Greiff, S. (2013). Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity. Learning and Individual Differences, 24, 42-52. doi: 10.1016/j.lindif.2012.12.011
59. Sonnleitner, P., Brunner, M., Greiff, S., Funke, J., Keller, U., Martin, R., . . . Latour, T. (2012). The Genetics Lab: Acceptance and psychometric characteristics of a computer-based microworld assessing complex problem solving. Psychological Test and Assessment Modeling, 54, 54-72.
60. Sonnleitner, P., Brunner, M., Keller, U., & Martin, R. (2014). Differential relations between facets of complex problem solving and students' immigration background. Journal of Educational Psychology, 106, 681-695. doi:10.1037/a0035506
61. Sonnleitner, P., Keller, U., Martin, R., & Brunner, M. (2013). Students' complex problem-solving abilities: Their structure and relations to reasoning ability and educational success. Intelligence, 41, 289-305. doi: 10.1016/j.intell.2013.05.002
62. Spitz-Oener, A. (2006). Technical change, job tasks and rising educational demands: Looking outside the wage structure. Journal of Labor Economics, 24, 235-270. doi:10.1086/499972
63. Süß, H.-M. (1996). Intelligenz, Wissen und Problemlösen: Kognitive Voraussetzungen für erfolgreiches Handeln bei computersimulierten Problemen [Intelligence, knowledge, and problem solving: Cognitive prerequisites of successful performance in computer-simulated problems]. Göttingen, Germany: Hogrefe.
64. Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 19-30). Cambridge, MA: Cambridge University Press. doi:10.1017/CBO9780511816819.003
65. Tsai, M.-J. (2002). Do male and female students often perform better than female students when learning computers? A study of Taiwanese eight graders' computer education through strategic and cooperative learning. Journal of Educational Computing Research, 26, 67-85. doi:10.2190/ 9JW6-VV1P-FAX8-CGE0
66. Van Braak, J. P. (2004). Domains and determinants of university students' self-perceived computer competence. Computers & Education, 43, 299-312. doi:10.1016/j.compedu.2003.09.006
67. Van der Linden, W. J., & Glas, C. A. W. (2000). Computerized adaptive testing. Dordrecht, the Netherlands: Kluwer Academic Publishers.
68. Vollmeyer, R., & Funke, J. (1999). Personen- und Aufgabenmerkmale beim komplexen Problemlösen [Person and task effects within complex problem solving]. Psychologische Rundschau, 50, 213-219. doi: 10.1026//0033-3042.50.4.213
69. Wenke, D., Frensch, P. A., & Funke, J. (2005). CPS and intelligence: Empirical relation and causal direction. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition and intelligence: Identifying the mechanisms of the mind (pp. 160-187). New York, NY: Cambridge University Press.
70. Williamson, D. M., Mislevy, R. J., & Bejar, I. I. (Eds.). (2006). Automated scoring of complex tasks in computer-based testing. Mahwah, NJ: Lawrence Erlbaum Associates.
71. Wirth, J., & Klieme, E. (2003). Computer-based assessment of problem solving competence. Assessment in Education: Principles, Policy & Practice, 10, 329-345. doi:10.1080/0969594032000148172
72. Wittmann, W. W., & Süß, H.-M. (1999). Investigating the paths between working memory, intelligence, knowledge, and complex problemsolving performances via Brunswik symmetry. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual differences: Process, trait, content determinants (pp. 77-108). Washington, DC: American Psychological Association. doi:10.1037/10315-004
73. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving-More than reasoning? Intelligence, 40, 1-14. doi:10.1016/j.intell .2011.11.003