More is not always better: The relation between item response and item response time in raven’s matrices

Journal of Intelligence

Volumn 3, Issue 1, 2015, Pages 21-40

  Goldhammer, Frank ^a   Naumann, Johannes ^b   Greiff, Samuel ^c  

^a GERMAN INSTITUTE FOR INTERNATIONAL EDUCATIONAL RESEARCH   (Germany)

^b GOETHE UNIVERSITY   (Germany)

^c UNIVERSITY OF LUXEMBOURG   (Luxembourg)

Author keywords

Error type; Generalized linear mixed modeling; Item response times; Moderation by item; Moderation by person; Number of rules; Reasoning

Indexed keywords

EID: 84949556097     PISSN: None     EISSN: 20793200     Source Type: Journal    
DOI: 10.3390/jintelligence3010021     Document Type: Article

References (53)

1. Van der Linden, W.J. A hierarchical framework for modeling speed and accuracy on test items. Psychometrika 2007, 72, 287–308.
2. Wickelgren, W.A. Speed-accuracy tradeoff and information processing dynamics. Acta Psychol. 1973, 41, 67–85.
3. Wise, S.L.; DeMars, C.E. An application of item response time: The effort-moderated IRT model. J. Educ. Meas. 2006, 43, 19–38.
4. Wise, S.L.; Kong, X. Response time effort: A new measure of examinee motivation in computer-based tests. Appl. Meas. Educ. 2005, 18, 163–183.
5. Heathcote, A.; Popiel, S.J.; Mewshort, D.J.K. Analysis of response time distributions: An example using the Stroop task. Psychol. Bull. 1991, 109, 340–347.
6. Miyake, A.; Friedman, N.P.; Emerson, M.J.; Witzki, A.H.; Howerter, A.; Wager, T.D. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognit. Psychol. 2000, 41, 49–100.
7. Goldhammer, F.; Naumann, J.; Stelter, A.; Tóth, K.; Rölke, H.; Klieme, E. The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. J. Educ. Psychol. 2014, 106, 608–626.
8. Verbić, S.; Tomić, B. Test item response time and the response likelihood. 2009. Available online: http://arxiv.org/ftp/arxiv/papers/0901/0901.4356.pdf (accessed on 15 August 2014).
9. Ebel, R.L. The use of item response time measurements in the construction of educational achievement tests. Educ. Psychol. Meas. 1953, 13, 391–401.
10. Hornke, L.F. Item response times in computerized adaptive testing. Psicológica 2000, 21, 175–189.
11. Carroll, J.B. Human Cognitive Abilities. A Survey of Factor-Analytical Studies; Cambridge University Press: New York, NY, USA, 1993.
12. Raven, J.C. Advanced Progressive Matrices; Lewis and Co. Ltd.: London, UK, 1962.
13. Klein Entink, R.H.; Fox, J.P.; van der Linden, W.J. A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika 2009, 74, 21–48.
14. Goldhammer, F.; Klein Entink, R.H. Speed of reasoning and its relation to reasoning ability. Intelligence 2011, 39, 108–119.
15. Van der Linden, W.J.; Scrams, D.J.; Schnipke, D.L. Using response-time constraints to control for differential speededness in computerized adaptive testing. Appl. Psychol. Meas. 1999, 23, 195–210.
16. Goldhammer, F.; Naumann, J.; Keßel, Y. Assessing individual differences in basic computer skills. Eur. J. Psychol. Assess. 2013, 29, 263–275.
17. Scherer, R.; Greiff, S.; Hautamäki, J. Exploring the relation between speed and ability in complex problem solving. Intelligence 2015, 48, 37–50.
18. Lasry, N.; Watkins, J.; Mazur, E.; Ibrahim, A. Response times to conceptual questions. Am. J. Phys. 2013, 81, 703.
19. Sporer, S.L. Eyewitness identification accuracy, confidence, and decision times in simultaneous and sequential lineups. J. Appl. Psychol. 1993, 78, 22–33.
20. Fitts, P.M.; Posner, M.I. Learning and Skilled Performance in Human Performance; Brooks/Cole: Belmont, CA, USA, 1967.
21. Schneider, W.; Shiffrin, R.M. Controlled and automatic human information-processing: 1. Detection, search, and attention. Psychol. Rev. 1977, 84, 1–66.
22. Schneider, W.; Chein, J.M. Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cognit. Sci. 2003, 27, 525–559.
23. Ackerman, P.L. Predicting individual differences in complex skill acquisition: Dynamics of ability determinants. J. Appl. Psychol. 1992, 77, 598–614.
24. Hornke, L.F. Response time in computer-aided testing: A “verbal memory” test for routes and maps. Psychol. Sci. 2005, 47, 280–293.
25. Beckmann, J.F. Differentielle Latenzzeiteffekte bei der Bearbeitung von Reasoning-Items. Diagnostica 2000, 46, 124–129. (In German)
26. Beckmann, J.F.; Beckmann, N. Effects of feedback on performance and response latencies in untimed reasoning tests. Psychol. Sci. 2005, 47, 262–278.
27. Neubauer, A.C. Speed of information processing in the Hick paradigm and response latencies in a psychometric intelligence test. Personal. Individ. Differ. 1990, 11, 147–152.
28. Dodonova, Y.A.; Dodonov, Y.S. Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty. Intelligence 2013, 41, 1–10.
29. Carpenter, P.A.; Just, M.A.; Shell, P. What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychol. Rev. 1990, 97, 404–431.
30. McGrew, K.S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence 2009, 37, 1–10.
31. Verguts, T.; de Boeck, P.; Maris, E. Generation speed in Raven’s progressive matrices test. Intelligence 1999, 27, 329–345.
32. Raven, J.C.; Raven, J.; Court, J.H. APM Manual (German Edition and Norming by H. Häcker and St. Bulheller); Swets Test Services: Frankfurt am Main, Germany, 1998.
33. Baayen, R.H.; Davidson, D.J.; Bates, D.M. Mixed-effects modeling with crossed random effects for subjects and items. J. Mem. Lang. 2008, 59, 390–412.
34. De Boeck, P.; Bakker, M.; Zwitser, R.; Nivard, M.; Hofman, A.; Tuerlinckx, F.; Partchev, I. The estimation of item response models with the lmer function from the lme4 package in R. J. Statist. Softw. 2011, 39, 1–28.
35. Doran, H.; Bates, D.; Bliese, P.; Dowling, M. Estimating the multilevel Rasch model: With the lme4 package. J. Statist. Softw. 2007, 20, 1–18.
36. De Boeck, P. Random item IRT models. Psychometrika 2008, 73, 533–559.
37. Van der Linden, W.J. Conceptual issues in response-time modeling. J. Educ. Meas. 2009, 46, 247–272.
38. Yen, W.M. Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Appl. Psychol. Meas. 1984, 8, 125–145.
39. Yen, W.M. Scaling performance assessments: Strategies for managing local item dependence. J. Educ. Meas. 1993, 30, 187–213.
40. Bolker, B.M.; Brooks, M.E.; Clark, C.J.; Geange, S.W.; Poulsen, J.R.; Stevens, M.H.; White, J.S. Generalized linear mixed models: A practical guide for ecology and evolution. Trends Ecol. Evol. 2009, 24, 127–135.
41. Greven, S.; Kneib, T. On the behaviour of marginal and conditional AIC in linear mixed models. Biometrika 2010, 97, 773–789.
42. Vaida, F.; Blanchard, S. Conditional Akaike information for mixed-effects models. Biometrika 2005, 92, 351–370.
43. Bates, D.; Maechler, M.; Bolker, B.; Walker, S. lme4: Linear Mixed-Effects Models Using Eigen and S4. 2013. Available online: http://cran.r-project.org/web/packages/lme4/citation.html (accessed on 11 March 2015).
44. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2013.
45. Roskam, E.E. Models for speed and time-limit tests. In Handbook of Modern Item Response Theory; van der Linden, W.J., Hambleton, R., Eds.; Springer: New York, NY, USA, 1997; pp. 87–208.
46. Wang, T.; Hanson, B.A. Development and calibration of an item response model that incorporates response time. Appl. Psychol. Meas. 2005, 29, 323–339.
47. Preckel, F. Diagnostik Intellektueller Hochbegabung. Testentwicklung zur Erfassung der Fluiden Intelligenz. Dissertation; Hogrefe: Göttingen, Germany, 2003.
48. Babcock, R.L. Analysis of age differences in types of errors on the Raven’s advanced progressive matrices. Intelligence 2002, 30, 485–503.
49. Vodegel Matzen, L.B.L.; van der Molen, M.W.; Dudink, A.C.M. Error analysis of Raven test performance. Personal. Individ. Differ. 1994, 16, 433–445.
50. Meo, M.; Roberts, M.J.; Marucci, F.S. Element salience as a predictor of item difficulty for Raven’s progressive matrices. Intelligence 2007, 35, 359–368.
51. Sutcliffe, A.; Rugg, G. A taxonomy of error types for failure analysis and risk assessment. Int. J. Hum. Comput. Interact. 1998, 10, 381–405.
52. Johnson-Laird, P.N. A model theory of induction. Int. Stud. Philos. Sci. 1994, 8, 5–29.
53. Johnson-Laird, P.N.; Girotto, V.; Legrenzi, P. Reasoning from inconsistency to consistency. Psychol. Rev. 2004, 111, 640–661.