The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment

Journal of Educational Psychology

Volumn 106, Issue 3, 2014, Pages 608-626

  Goldhammer, Frank ^a,b   Naumann, Johannes ^a,b   Stelter, Annette ^a   Tóth, Krisztina ^a   Rölke, Heiko ^a   Klieme, Eckhard ^a,b  

^a GERMAN INSTITUTE FOR INTERNATIONAL EDUCATIONAL RESEARCH   (Germany)

^b Centre for International Student Assessment (ZIB)   (Germany)

Author keywords

Automatic and controlled processing; Computer based assessment; Problem solving; Reading literacy; Time on task

Indexed keywords

EID: 84894285824     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0034716     Document Type: Article

References (80)

1. Ackerman, P. L. (1987). Individual differences in skill learning: An integration of psychometric and information processing perspectives. Psychological Bulletin, 102, 3-27. doi:10.1037/0033-2909.102.1.3
2. Ackerman, P. L., & Cianciolo, A. T. (2000). Cognitive, perceptual speed, and psychomotor determinants of individual differences during skill acquisition. Journal of Experimental Psychology: Applied, 6, 259-290. doi:10.1037/1076-898X.6.4.259
3. Anderson, J. R. (1992). Automaticity and the ACTß theory. American Journal of Psychology, 105, 165-180. doi:10.2307/1423026
4. Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, NJ: Erlbaum.
5. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412. doi:10.1016/j.jml.2007.12.005
6. Balota, D. A., Cortese, M. J., Sergent-Marshall, S., Spieler, D. H., & Yap, M. J. (2004). Visual word recognition of single-syllable words. Journal of Experimental Psychology: General, 133, 283-316. doi:10.1037/0096- 3445.133.2.283
7. Bates, D., Maechler, M., & Bolker, B. (2012). lme4: Linear mixed-effects models using Eigen and S4 (R Package Version 0.999999-0) [Computer software]. Retrieved from http://CRAN.R-project.org/ package=lme4
8. Baumert, J., Evans, R. H., & Geiser, H. (1998). Technical problem solving among 10-year-old students as related to science achievement, out-ofschool experience, domain-specific control beliefs, and attribution patterns. Journal of Research in Science Teaching, 35, 987-1013. doi: 10.1002/(SICI)1098-2736(199811)35:9987::AID-TEA33.0.CO;2-P
9. Beckmann, J. F. (2000). Differentielle Latenzzeiteffekte [Differential effects of latencies]. Diagnostica, 46, 124-129. doi:10.1026//0012-1924 .46.3.124
10. Best, R., Rowe, M., Ozuru, Y., & McNamara, D. (2005). Deep-level comprehension of science texts: The role of the reader and the text. Topics in Language Disorders, 25, 65-83. doi:10.1097/00011363- 200501000-00007
11. Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology and Evolution, 24, 127-135. doi:10.1016/j.tree.2008.10.008
12. Carlson, R. A., Khoo, B. H., Yaure, R. G., & Schneider, W. (1990). Acquisition of a problem-solving skill: Levels of organization and use of working memory. Journal of Experimental Psychology: General, 119, 193-214.
13. Carlson, R. A., Sullivan, M. A., & Schneider, W. (1989). Component fluency in a problem-solving context. Human Factors, 31, 489-502. doi:10.1177/001872088903100501
14. Carver, R. P. (1992). Reading rate: Theory, research and practical implications. Journal of Reading, 36, 84-95.
15. De Boeck, P. (2008). Random item IRT models. Psychometrika, 73, 533-559. doi:10.1007/s11336-008-9092-x
16. De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39, 1-28.
17. De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer. doi:10.1007/978-1-4757-3990-9
18. Dodonova, Y. A., & Dodonov, Y. S. (2013). Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty. Intelligence, 41, 1-10. doi:10.1016/j.intell.2012.10 .003
19. Doran, H., Bates, D., Bliese, P., & Dowling, M. (2007). Estimating the multilevel Rasch model: With the lme4 package. Journal of Statistical Software, 20, 1-18.
20. Ebel, R. (1953). The use of item response time measurements in the construction of educational achievement tests. Educational and Psychological Measurement, 13, 391-401. doi:10.1177/001316445301300303
21. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4, 19-41. doi:10.7771/1932- 6246.1118
22. Fitts, P. M., & Posner, M. I. (1967). Learning and skilled performance in human performance. Belmont, CA: Brooks/Cole.
23. Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7, 69-89. doi:10.1080/13546780042000046
24. Funke, J., & Frensch, P. A. (2007). Complex problem solving: The European perspective-10 years after. In D. H. Jonassen (Ed.), Learning to solve complex scientific problems (pp. 25-47). New York, NY: Erlbaum.
25. Gelman, A. (2005). Analysis of variance? Why it is more important than ever. Annals of Statistics, 33, 1-53. doi:10.1214/009053604000001048
26. Goldhammer, F., & Klein Entink, R. H. (2011). Speed of reasoning and its relation to reasoning ability. Intelligence, 39, 108-119. doi:10.1016/j .intell.2011.02.001
27. Goldhammer, F., Naumann, J., & Keßel, Y. (2013). Assessing individual differences in basic computer skills: Psychometric characteristics of an interactive performance measure. European Journal of Psychological Assessment. Advance online publication. doi:10.1027/1015-5759/ a000153
28. Gräsel, C., Fischer, F., & Mandl, H. (2000). The use of additional information in problem-oriented learning environments. Learning Environments Research, 3, 287-305. doi:10.1023/A:1011421732004
29. Greiff, S., Holt, D. V., & Funke, J. (2013). Perspectives on problem solving in educational assessment: Analytical, interactive, and collaborative problem solving. Journal of Problem Solving, 5, 71-91. doi:10.7771/ 1932-6246.1153
30. Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapó, B. (2013). Complex problem solving in educational settings-Something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105, 364 -379. doi: 10.1037/a0031856
31. Guthrie, J. T., Wigfield, A., Barbosa, P., Perencevich, K. C., Taboada, A., Davis, M. H., . . . Tonks, S. (2004). Increasing reading comprehension and engagement through Concept-Oriented Reading Instruction. Journal of Educational Psychology, 96, 403-423. doi:10.1037/0022-0663.96.3 .403
32. Hornke, L. F. (2000). Item response times in computerized adaptive testing. Psicológica, 21, 175-189.
33. Hornke, L. F. (2005). Response time in computer-aided testing: A "verbal memory" test for routes and maps. Psychology Science, 47, 280-293.
34. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, England: Cambridge University Press.
35. Klein Entink, R. H., Fox, J.-P., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21-48. doi:10.1007/s11336-008- 9075-y
36. Klieme, E. (2004). Assessment of cross-curricular problem-solving competencies. In J. H. Moskowitz & M. Stephens (Eds.), Comparing learning outcomes: International assessments and education policy (pp. 81-107). London, England: Routledge.
37. Krajewski, K., & Schneider, W. (2009). Early development of quantity to number-word linkage as a precursor of mathematical school achievement and mathematical difficulties: Findings from a four-year longitudinal study. Learning and Instruction, 19, 513-526. doi:10.1016/j .learninstruc.2008.10.002
38. Kupiainen, S., Vainikainen, M.-P., Marjanen, J., & Hautamäki, J. (2014). The role of time on task in computer-based assessment of crosscurricular skills. Journal of Educational Psychology, 106, 627-638. doi:10.1037/a0035507
39. Landerl, K., & Wimmer, H. (2008). Development of word reading fluency and spelling in a consistent orthography: An 8-year follow-up. Journal of Educational Psychology, 100, 150-161. doi:10.1037/0022-0663.100 .1.150
40. Luce, R. D. (1986). Response times: Their roles in inferring elementary mental organization. Oxford, England: Oxford University Press.
41. Mayer, R. E. (1992). Thinking, problem solving, cognition (2nd ed.). New York, NY: Freeman.
42. Mayer, R. E. (1994). Problem solving, teaching and testing for. In T. Husén & T. N. Postlethwaite (Eds.), The international encyclopedia of education (2nd ed., Vol. 8, pp. 4728-4731). Oxford, England: Pergamon Press.
43. McKeown, M. G., Beck, I. L., & Blake, R. G. K. (2009). Rethinking reading comprehension instruction: A comparison of instruction for strategies and content approaches. Reading Research Quarterly, 44, 218-253. doi:10.1598/RRQ.44.3.1
44. Morrell, C. H., Pearson, J. D., & Brant, L. J. (1997). Linear transformations of linear mixed-effects models. American Statistician, 51, 338-343. doi:10.1080/00031305.1997.10474409
45. Naumann, J., Richter, T., Christmann, U., & Groeben, N. (2008). Working memory capacity and reading skill moderate the effectiveness of strategy training in learning from hypertext. Learning and Individual Differences, 18, 197-213. doi:10.1016/j.lindif.2007.08.007
46. Naumann, J., Richter, T., Flender, J., Christmann, U., & Groeben, N. (2007). Signaling in expository hypertexts compensates for deficits in reading skill. Journal of Educational Psychology, 99, 791-807. doi: 10.1037/0022-0663.99.4.791
47. Neubauer, A. C. (1990). Speed of information processing in the Hick paradigm and response latencies in a psychometric intelligence test. Personality and Individual Differences, 11, 147-152. doi:10.1016/0191- 8869(90)90007-E
48. Nicolson, R. I., & Fawcett, A. J. (1994). Reaction times and dyslexia. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 47(A), 29-48. doi:10.1080/14640749408401142
49. Organisation for Economic Co-Operation and Development. (2009). PIAAC literacy: A conceptual framework (OECD Education Working Paper No. 34). Paris, France: Author.
50. Organisation for Economic Co-Operation and Development. (2009). PIAAC problem solving in technology-rich environments: A conceptual framework (OECD Education Working Paper No. 36). Paris, France: Author.
51. Organisation for Economic Co-Operation and Development. (2011). PISA 2009 results: Vol. VI. Digital technologies and performance. Paris, France: Author.
52. Organisation for Economic Co-Operation and Development. (2013). OECD skills outlook 2013: First results from the Survey of Adult Skills. doi:10.1787/9789264204256-en
53. Perfetti, C. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11, 357-383. doi:10.1080/ 10888430701530730
54. Puntambekar, S., & Stylianou, A. (2005). Designing navigation support in hypertext systems based on navigation patterns. Instructional Science, 33, 451-481. doi:10.1007/s11251-005-1276-5
55. R Core Team. (2012). R: A language and environment for statistical computing [Computer software]. Retrieved from http://www.R-project. org/
56. Richter, T., Isberner, M.-B., Naumann, J., & Kutzner, Y. (2012). Prozessbezogene Diagnostik von Lesefähigkeiten bei Grundschulkindern [Process-based measurement of reading skills in primary school children]. Zeitschrift für Pädagogische Psychologie/German Journal of Educational Psychology, 26, 313-331. doi:10.1024/1010-0652/a000079
57. Richter, T., Isberner, M.-B., Naumann, J., & Neeb, Y. (2013). Lexical quality and reading comprehension in primary school children. Scientific Studies of Reading. Advance online publication. doi:10.1080/10888438 .2013.764879
58. Richter, T., Naumann, J., Brunner, M., & Christmann, U. (2005). Strategische Verarbeitung beim Lernen mit Text und Hypertext [Strategic processing in learning from text and hypertext]. Zeitschrift für Pädagogische Psychologie/German Journal of Educational Psychology, 19, 5-22. doi:10.1024/1010-0652.19.12.5
59. Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden & R. Hambleton (Eds.), Handbook of modern item response theory (pp. 187-208). New York, NY: Springer. doi:10.1007/978-1- 4757-2691-6_11
60. Rouet, J.-F., & Passerault, J.-M. (1999). Analyzing learner-hypermedia interaction: An overview of on-line methods. Instructional Science, 27, 201-219. doi:10.1007/BF00897319
61. Scheuermann, F., & Björnsson, J. (2009). The transition to computer-based assessment. Luxembourg: Office for Official Publications of the European Communities.
62. Schleicher, A. (2008). PIAAC: A new strategy for assessing adult competencies. International Review of Education, 54, 627-650. doi:10.1007/ s11159-008-9105-0
63. Schneider, W., & Chein, J. M. (2003). Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cognitive Science, 27, 525-559. doi:10.1016/S0364-0213(03)00011-9
64. Schneider, W., & Fisk, A. D. (1983). Attentional theory and mechanisms for skilled performance. In R. A. Magill (Ed.), Memory and control of action (pp. 119-143). New York, NY: North-Holland. doi:10.1016/ S0166-4115(08)61989-5
65. Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84, 1-66. doi:10.1037/0033-295X.84.1.1
66. Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190. doi:10.1037/0033- 295X.84.2.127
67. Simon, H. A., & Newell, A. (1971). Human problem solving: The state of the theory in 1970. American Psychologist, 26, 145-159. doi:10.1037/ h0030806
68. Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. M. Downing & T. S. Haladyna (Eds.), Handbook of test development (pp. 329-347). Mahwah, NJ: Erlbaum.
69. Sullivan, S., Gnesdilow, D., & Puntambekar, S. (2011). Navigation behaviors and strategies used by middle school students to learn from a science hypertext. Journal of Educational Multimedia and Hypermedia, 20, 387-423.
70. van den Broek, P. W., & Espin, C. A. (2012). Connecting cognitive theory and assessment: Measuring individual differences in reading comprehension. School Psychology Review, 41, 315-325.
71. van der Linden, W. J. (2005). Linear models for optimal test design. New York, NY: Springer.
72. van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287-308. doi: 10.1007/s11336-006-1478-z
73. van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247-272. doi:10.1111/j .1745-3984.2009.00080.x
74. van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195-210. doi:10.1177/01466219922031329
75. Walczyk, J. (2000). The interplay between automatic and control processes in reading. Reading Research Quarterly, 35, 554-566. doi:10.1598/ RRQ.35.4.7
76. Wickelgren, W. A. (1977). Speed-accuracy tradeoff and information processing dynamics. Acta Psychologica, 41, 67-85. doi:10.1016/0001- 6918(77)90012-9
77. Wirth, J., & Klieme, E. (2003). Computer-based assessment of problem solving competence. Assessment in Education: Principles, Policy & Practice, 10, 329-345. doi:10.1080/0969594032000148172
78. Wirth, J., & Leutner, D. (2008). Self-regulated learning as a competence: Implications of theoretical models for assessment methods. Zeitschrift für Psychologie/Journal of Psychology, 216, 102-110. doi:10.1027/ 0044-3409.216.2.102
79. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving: More than reasoning? Intelligence, 40, 1-14. doi:10.1016/j.intell.2011 .11.003
80. Zoanetti, N. (2010). Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology, 26, 585-606.