Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files

Computers in Human Behavior

Volumn 61, Issue , 2016, Pages 36-46

  Greiff, Samuel ^a   Niepel, Christoph ^a   Scherer, Ronny ^b   Martin, Romain ^a  

^a UNIVERSITY OF LUXEMBOURG   (Luxembourg)

^b UNIVERSITY OF OSLO   (Norway)

Author keywords

Complex problem solving; Computer based assessment; Log file data; Nonlinear relations; Time on task; VOTAT

Indexed keywords

COMPUTER AIDED ANALYSIS; EDUCATION; STUDENTS;

COMPLEX PROBLEM SOLVING; COMPUTER-BASED ASSESSMENTS; LOG FILE; NONLINEAR RELATIONS; TIME ON TASKS; VOTAT;

PROBLEM SOLVING;

ADOLESCENT; BEHAVIOR; EXECUTIVE FUNCTION TEST; HUMAN; MAJOR CLINICAL STUDY; PROBLEM SOLVING; STUDENT;

EID: 84960155777     PISSN: 07475632     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.chb.2016.02.095     Document Type: Article

References (48)

1. D. Albert, and L. Steinberg Age differences in strategic planning as indexed by the Tower of London Child Development 82 2011 1501 1517
2. X.S. Apedoe, and C.D. Schunn Strategies for success: uncovering what makes students successful in design and learning Instructional Science 41 2013 773 791
3. T. Asparouhov, and B. Muthén Weighted least squares estimation with missing data 2010 Muthén & Muthén Los Angeles, CA Retrieved from: statmodel.com/download/GstrucMissingRevision.pdf (accessed 22/11/2015)
4. A. Beauducel, and P.Y. Herzberg On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA Structural Equation Modeling 13 2006 186 203
5. C.V. Bunderson, D.K. Inouye, and J.B. Olsen The four generations of computerized educational measurement R.L. Linn, Educational measurement 1989 Macmillan Publishing New York, NY 367 407
6. S. van Buuren, and K. Goorthius-Oudshoorn Multivariate imputation by chained equations (mice). R package version 2.25 2015 cran.r-project.org/web/packages/mice/index.html
7. S. Croker, and H. Buchanan Scientific reasoning in a real-world context: The effect of prior belief and outcome on children's hypothesis-testing strategies British Journal of Developmental Psychology 29 2011 409 424
8. Y. Dodonova, and Y. Dodonov Processing speed and intelligence as predictors of school achievement: mediation or unique contribution? Intelligence 40 2012 163 171
9. D. Dörner On the difficulties people have in dealing with complexity Simulation & Gaming 11 1980 87 106
10. D. Dörner, and H. Schaub Errors in planning and decision-making and the nature of human information processing Applied Psychology 43 1994 433 453
11. C. Enders Applied missing data analysis 2010 The Guilford Press NewYork, NY
12. A. Fischer, S. Greiff, and J. Funke The process of solving complex problems The Journal of Problem Solving 4 2012 19 41
13. P.A. Frensch, and J. Funke Definitions, traditions, and a general framework for understanding complex problem solving P.A. Frensch, J. Funke, Complex problem solving: The European perspective 1995 Lawrence Erlbaum Associates Hillsdale, NJ 3 25
14. J. Funke Dynamic systems as tools for analysing human judgement Thinking and Reasoning 7 2001 69 89
15. F. Goldhammer, J. Naumann, and S. Greiff More is not always better. The relation between response and response time and their moderation by item and person characteristics in Raven's matrices Journal of Intelligence 3 2015 21 40
16. F. Goldhammer, J. Naumann, A. Stelter, K. Tóth, H. Rölke, and E. Klieme The time on task effect in reading and problem solving is moderated by task difficulty and skill: insights from a computer-based large-scale assessment Journal of Educational Psychology 106 2014 608 626
17. S. Greiff, A. Fischer, S. Wüstenberg, P. Sonnleitner, M. Brunner, and R. Martin A multitrait-multimethod study of assessment instruments for complex problem solving Intelligence 41 2013 579 596
18. S. Greiff, D.V. Holt, and J. Funke Perspectives on problem solving in cognitive research and educational assessment: Analytical, interactive, and collaborative problem solving Journal of Problem Solving 5 2013 71 91
19. S. Greiff, A. Kretzschmar, J.C. Neubert, B. Spinath, and R. Martin Computer-based assessment of complex problem solving and how it is influenced by students' information and communication technology literacy Journal of Educational Psychology 106 2014 666 680
20. S. Greiff, S. Wüstenberg, and F. Avvisati Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving Computers & Education 91 2015 92 105
21. S. Greiff, S. Wüstenberg, B. Csapo, A. Demetriou, J. Hautamäki, A.C. Graesser, and et al. Domain-general problem solving skills and education in the 21st century Educational Research Review 13 2014 74 83
22. S. Greiff, S. Wüstenberg, and J. Funke Dynamic problem solving: A new measurement perspective Applied Psychological Measurement 36 2012 189 213
23. L. Hu, and P.M. Bentler Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives Structural Equation Modeling 6 1999 1 55
24. D.H. Jonassen Learning to solve problems 2011 Routledge New York, NY
25. A. Kluge Performance assessment with microworlds and their difficulty Applied Psychological Measurement 32 2008 156 180
26. S. Kröner, J.L. Plass, and D. Leutner Intelligence assessment with computer simulations Intelligence 33 2005 347 368
27. D. Kuhn, and D. Dean Is developing scientific thinking all about learning to control variables? Psychological Science 16 2005 866 870
28. S. Kupiainen, M. Vainikainen, J. Marjanen, and J. Hautamäki The role of time on task in computer-based low-stakes assessment of cross-curricular skills Journal of Educational Psychology 106 2014 627 638
29. T.D. Little, J.A. Bovaird, and K.F. Widaman On the merits of orthogonalizing powered and product terms: implications for modeling interactions among latent variables Structural Equation Modeling 13 2006 497 519
30. H. Marsh, K. Hau, and D. Grayson Goodness of fit evaluation in structural equation modeling A. Maydeu-Olivares, J. McArdle, Contemporary psychometrics 2005 Erlbaum Mahwah, NJ 275 340
31. R. Martin, and C. Houssemand Interets et limites de la chronometrie mentale dans la mesure psychologique Interests and limits of the measurement of time in psychological assessment Bulletin De Psychologie 55 2002 605 614
32. R.E. Mayer, and M.C. Wittrock Problem solving Handbook of educational psychology 2 2006 287 303
33. L.K. Muthén, and B.O. Muthén Mplus user's guide. Version 7.11 1998-2013 Muthén & Muthén Los Angeles, CA
34. L.R. Novick, and M. Bassok Problem solving K.J. Holyoak, R.G. Morrison, The Cambridge handbook of thinking and reasoning 2005 University Press Cambridge, NY 321 349
35. OECD Programme for international student assessment: An overview 2009 OECD Paris
36. OECD PISA 2012 results: Creative problem solving 2014 OECD Publishing Paris
37. M. Osman Controlling uncertainty: A review of human behavior in complex dynamic environments Psychological Bulletin 136 2010 65 86
38. J. Raven Psychometrics, cognitive ability, and occupational performance Review of Psychology 7 2000 51 74
39. R. Scherer, S. Greiff, and J. Hautamäki Exploring the relation between time on task and ability in complex problem solving Intelligence 48 2015 37 50
40. R. Scherer, and J.E. Gustafsson The relations among openness, perseverance, and performance in creative problem solving: A substantive-methodological approach Thinking Skills and Creativity 18 2015 4 17
41. F. Schweizer, S. Wüstenberg, and S. Greiff Validity of the MicroDYN approach: complex problem solving predicts school grades beyond working memory capacity Learning and Individual Differences 24 2013 42 52
42. H. Thillmann, J. Gößling, J. Marschner, J. Wirth, and D. Leutner Metacognitive knowledge about and metacognitive regulation of strategy use in self-regulated discovery learning: new methods of assessment in computer-based learning environments R. Azevedo, V. Aleven, International handbook of metacognition and learning technologies 2013 Springer New York 575 588
43. J.E. Tschirgi Sensible reasoning: A hypothesis about hypotheses Child Development 51 1980 1 10
44. M.-P. Vainikainen Finnish primary school pupils' performance in learning to learn assessments: A longitudinal perspective on educational equity University of Helsinki, Department of Teacher Education Research Reports, 360 2014 Picaset Helsinki
45. R. Vollmeyer, B.D. Burns, and K.J. Holyoak The impact of goal specificity on strategy use and the acquisition of problem structure Cognitive Science 20 1996 75 100
46. S. Wüstenberg, S. Greiff, and J. Funke Complex problem solving - more than reasoning? Intelligence 40 2012 1 14
47. S. Wüstenberg, M. Stadler, J. Hautamäki, and S. Greiff The role of strategy knowledge for the application of strategies in complex problem solving tasks Technology, Knowledge and Learning 19 2014 127 146
48. R.E. Zinbarg, W. Revelle, I. Yovel, and W. Li Cronbach's α, Revelle's β, and McDonald's ω: their relations with each other and two alternative conceptualizations of reliability Psychometrika 70 2005 123 133