Evaluating Cognitive Theory: A Joint Modeling Approach Using Responses and Response Times

Psychological Methods

Volumn 14, Issue 1, 2009, Pages 54-75

  Entink, Rinke H Klein ^a   Kuhn, Jörg Tobias ^b   Hornke, Lutz F ^c   Fox, Jean Paul ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

^b UNIVERSITY OF MÜNSTER   (Germany)

^c RWTH AACHEN UNIVERSITY   (Germany)

Author keywords

item response theory; response times; rule based item design

Indexed keywords

APTITUDE TEST; ARTICLE; ATTENTION; BAYES THEOREM; COGNITION; CONCEPT FORMATION; EDUCATION; HUMAN; PATTERN RECOGNITION; PROBLEM SOLVING; PSYCHOLOGIC TEST; PSYCHOMETRY; REACTION TIME; REPRODUCIBILITY; STATISTICAL MODEL; STATISTICS;

APTITUDE TESTS; ATTENTION; BAYES THEOREM; COGNITION; CONCEPT FORMATION; EDUCATIONAL MEASUREMENT; HUMANS; MODELS, STATISTICAL; PATTERN RECOGNITION, VISUAL; PROBLEM SOLVING; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REACTION TIME; REPRODUCIBILITY OF RESULTS;

EID: 64649103349     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0014877     Document Type: Article

References (88)

1. Adams R.J., Wilson M., and Wang W.-C. The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement 21 (1997) 1-23
2. Albert J.H. Bayesian estimation of normal ogive item response curves using Gibbs sampling. Journal of Educational Statistics 17 (1992) 251-269
3. Anderson T.W. An introduction to multivariate statistical analysis. 2nd ed. (1984), Wiley, New York
4. Bejar I.I., and Yocom P. A generative approach to the modeling of isomorphic hidden-figure items. Applied Psychological Measurement 15 (1991) 129-137
5. Bethell-Fox C.E., Lohman D.F., and Snow R.E. Adaptive reasoning: Componential and eye movement analysis of geometric analogy performance. Intelligence 8 (1984) 205-238
6. Borsboom D., Mellenbergh G.J., and van Heerden J. The theoretical status of latent variables. Psychological Review 110 (2003) 203-219
7. Borsboom D., Mellenbergh G.J., and van Heerden J. The concept of validity. Psychological Review 111 (2004) 1061-1071
8. Box G.E.P., and Tiao G.C. Bayesian inference in statistical analysis (1973), Addison-Wesley, Reading, MA
9. Carpenter P.A., Just M.A., and Shell P. What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychological Review 97 (1990) 404-431
10. Casella G., and Berger R.L. Statistical inference. 2nd ed. (2002), Duxbury, Pacific Grove, CA
11. De Jong M.G., Steenkamp J.-B.E.M., and Fox J.-P. Relaxing measurement invariance in cross-national consumer research using a hierarchical IRT model. Journal of Consumer Research 34 (2007) 260-278
12. Dickey J.M. The weighted likelihood ratio, linear hypothesis on normal location parameters. The Annals of Mathematical Statistics 42 (1971) 204-223
13. Dzhafarov E.N., and Schweickert R. Decompositions of response times: An almost general theory. Journal of Mathematical Psychology 39 (1995) 285-314
14. Edwards J.R., and Bagozzi R.P. On the nature and direction of relationships between constructs and measures. Psychological Methods 5 (2000) 155-174
15. Embretson S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 93 (1983) 179-197
16. Embretson S.E. Multicomponent response models. In: van der Linden W.J., and Hambleton R.K. (Eds). Handbook of modern item response theory (1997), Springer, New York 305-321
17. Embretson S.E. A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods 3 (1998) 380-396
18. Embretson S.E. Generating items during testing: Psychometric issues and models. Psychometrika 64 (1999) 407-433
19. Embretson S.E., and Reise S.P. Item response theory for psychologists (2000), Erlbaum, Mahwah, NJ
20. Engle R.W., Tuholski S.W., Laughlin J.E., and Conway A.R.A. Working memory, short-term memory, and general fluid intelligence: A latent variable approach. Journal of Experimental Psychology: General 128 (1999) 309-331
21. Ferrando P.J., and Lorenzo-Seva U. A measurement model for Likert responses that incorporates response time. Multivariate Behavioral Research 42 (2007) 675-706
22. Fischer G.H. The linear logistic test model as an instrument in educational research. Acta Psychologica 37 (1973) 359-374
23. Fox J.-P., and Glas C.A.W. Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika 66 (2001) 269-286
24. Fox J.-P., Klein Entink R.H., and van der Linden W.J. Modeling of responses and response times with the package cirt. Journal of Statistical Software 20 (2007) 1-14
25. Gelman A., Carlin J.B., Stern H.S., and Rubin D.B. Bayesian data analysis. 2nd ed. (2004), Chapman & Hall/CRC, New York
26. Gelman A., Meng X.-L., and Stern H. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica 6 (1996) 733-807
27. Gelman A., and Pardoe I. Bayesian measures of explained variance and pooling in multilevel (hierarchical) models. Technometrics 48 (2006) 241-251
28. Geman S., and Geman D. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (1984) 721-741
29. Glas C.A.W., and van der Linden W.J. Computerized adaptive testing with item cloning. Applied Psychological Measurement 23 (2003) 249-263
30. Gorin J.S. Manipulating processing difficulty of reading comprehension items: The feasibility of verbal item generation. Journal of Educational Measurement 42 (2005) 351-373
31. Gorin J.S. Test design with cognition in mind. Educational Measurement: Issues and Practice 25 (2006) 21-35
32. Holden R.R., and Kroner D.G. Relative efficacy of differential response latencies for detecting faking on a self-report measure of psychopathology. Psychological Assessment 4 (1992) 170-173
33. Holland P.W., and Rosenbaum P.R. Conditional association and unidimensionality in monotone latent variable models. The Annals of Statistics 14 (1986) 1523-1543
34. Hornke L.F. Item-generation models for higher order cognitive functions. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Erlbaum, Mahwah, NJ 159-178
35. Hornke L.F., and Habon M.W. Rule-based item bank construction and evaluation within the linear logistic framework. Applied Psychological Measurement 10 (1986) 369-380
36. Hornke L.F., and Wilding U. Konstanz von Itemparametern bei parallelen Itembanken [Constancy of item parameters in parallel item banks] (Tech. Rep.) (1997) RWTH Aachen University
37. Irvine S.H. The foundations of item generation for mass testing. In: Irvine S.H., and Kyllonen P.C. (Eds). Item generation for test development (2002), Erlbaum, Mahwah, NJ 3-34
38. Jacobs P.I., and Vandeventer M. Evaluating the teaching of intelligence. Educational and Psychological Measurement 32 (1972) 235-248
39. Janssen R., Schepers J., and Peres D. Models with item and item group predictors. In: DeBoeck P., and Wilson M. (Eds). Explanatory item response models: A generalized linear and nonlinear approach (2004), Springer, New York 189-212
40. Junker B.W., and Sijtsma K. Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement 25 (2001) 258-272
41. Kass R.E., and Raftery A.E. Bayes factors. Journal of the American Statistical Association 90 (1995) 773-795
42. Klein Entink, R. H., Fox, J.-P., & van der Linden, W. J. (in press). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika. (DOI: 10.1007/S11336-008- 9075-Y).
43. Klugkist I., Laudy O., and Hoijtink H. Inequality constrained analysis of variance: A Bayesian approach. Psychological Methods 10 (2005) 477-493
44. Lanza S.T., Collins L.M., Schafer J.L., and Flaherty B.P. Using data augmentation to obtain standard errors and conduct hypothesis tests in latent class and latent transition analysis. Psychological Methods 10 (2005) 84-100
45. Leighton J.P., and Gierl M.J. Cognitive diagnostic assessment for education: Theory and applications (2007), Cambridge University Press, New York
46. Lord F.M. Applications of item response theory to practical testing items (1980), Erlbaum, Hillsdale, NJ
47. Lord F.M., and Novick M.R. Statistical theories of mental test scores (1968), Addison-Wesley, Reading, MA
48. Luce D.R. Response times: Their role in inferring elementary mental organization (1986), Oxford University Press, New York
49. Maris E. Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times. Psychometrika 58 (1993) 445-469
50. McDonald R.P. The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology 34 (1981) 100-117
51. Mislevy R.J. Cognitive psychology and educational assessment. In: Brennan R.L. (Ed). Educational measurement. 4th ed. (2006), American Council on Education/Praeger, Westport, CT 257-305
52. Mislevy R.J., and Verhelst N. Modeling item responses when different subjects employ different solution strategies. Psychometrika 55 (1990) 195-215
53. Mulholland T.M., Pellegrino J.W., and Glaser R. Components of geometric analogy solution. Cognitive Psychology 12 (1980) 252-284
54. Newell A., and Simon H.A. Human problem solving (1972), Prentice-Hall, Englewood Cliffs, NJ
55. Patz R.J., and Junker B.W. A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics 24 (1999) 146-178
56. Primi R. Complexity of geometric inductive reasoning tasks: Contribution to the understanding of fluid intelligence. Intelligence 30 (2001) 41-70
57. Rasch G. Probabilistic models for some intelligence and attainment tests (1960), Danmarks Paedogogische Institut, Copenhagen, Denmark
58. Ratcliff R. A theory of memory retrieval. Psychological Review 85 (1978) 59-108
59. Raven J.C. Advanced progressive matrices, Set II (1962), H. K. Lewis, London
60. Rijmen F., and De Boeck P. The random weights linear logistic test model. Applied Psychological Measurement 26 (2002) 271-285
61. Robert C.P., and Casella G. Monte Carlo statistical methods (2004), Springer, New York
62. Roskam E.E. Models for speed and time-limit tests. In: van der Linden W.J., and Hambleton R. (Eds). Handbook of modern item response theory (1997), Springer, New York 187-208
63. Rost J. Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement 14 (1990) 271-282
64. Rouder J.N., Lu J., Morey R.D., Sun D., and Speckman P.L. A hierarchical process dissociation model. Journal of Experimental Psychology: General 137 (2008) 370-389
65. Rouder J.N., Sun D., Speckman P.L., Lu J., and Zhou D. A hierarchical Bayesian statistical framework for response time distributions. Psychometrika 68 (2003) 589-606
66. Rupp A.A., and Mislevy R.J. Cognitive foundations of structured item response models. In: Leighton J.P., and Gierl M.J. (Eds). Cognitive diagnostic assessment for education: Theories and applications (2007), Cambridge University Press, Cambridge, England 205-240
67. Scheiblechner H. Specific objective stochastic latency mechanisms. Journal of Mathematical Psychology 19 (1979) 18-38
68. Schmiedek F., Oberauer K., Wilhelm O., Süss H.-M., and Wittmann W.W. Individual differences in components of reaction time distributions and their relations to working memory and intelligence. Journal of Experimental Psychology: General 136 (2007) 414-429
69. Schnipke D.L., and Scrams D.J. Modeling item response times with a two-state mixture model: A new method for measuring speededness. Journal of Educational Measurement 34 (1997) 213-232
70. Sinharay S. Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement 42 (2005) 375-394
71. Sinharay S., Johnson M.S., and Stern H.S. Posterior predictive assessment of item response theory models. Applied Psychological Measurement 30 (2006) 298-321
72. Smith B.J. Boa: An R package for MCMC output convergence assessment and posterior inference. Journal of Statistical Software 21 (2007) 1-37
73. Spieler D.H., Balota D.A., and Faust M.E. Levels of selective attention revealed through analyses of response time distributions. Journal of Experimental Psychology: Human Perception and Performance 26 (2000) 506-526
74. Sternberg R.J. Component processes in analogical reasoning. Psychological Review 84 (1977) 353-378
75. Thissen D. Timed testing: An approach using item response theory. In: Weiss D.J. (Ed). New horizons in testing (1983), Academic Press, New York 179-203
76. van Breukelen G.J.P. Psychometric and information processing properties of selected response time models. Psychometrika 60 (1995) 95-113
77. van Breukelen G.J.P. Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika 70 (2005) 359-376
78. van der Linden W.J. A lognormal model for response times on test items. Journal of Educational and Behavioural Statistics 31 (2006) 181-204
79. van der Linden W.J. A hierarchical framework for modeling speed and accuracy on test items. Psychometrika 72 (2007) 287-308
80. van der Linden, W. J., & Guo, F. (in press). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika
81. van der Linden W.J., Scrams D.J., and Schnipke D.L. Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement 23 (1999) 195-210
82. van der Linden W.J., and van Krimpen-Stoop E.M.L.A. Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika 68 (2003) 251-265
83. van Zandt T. Analysis of response time distributions. In: Pashler H., and Wixted J. (Eds). Steven's handbook of experimental psychology, Vol. 4: Methodology in experimental psychology. 3rd ed. (2002), Wiley, New York 461-516
84. Verdinelli I., and Wasserman L. Computing Bayes factors using a generalization of the Savage-Dickey density ratio. Journal of the American Statistical Association 90 (1995) 614-618
85. Verhelst N., Verstralen H., and Jansen M. Models for time-limit tests. In: van der Linden W.J., and Hambleton R.K. (Eds). Handbook of modern item response theory (1997), Springer, New York 169-185
86. Wang T., and Hanson B.A. Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement 29 (2005) 323-339
87. Ward J., and Fitzpatrick F. Characteristics of matrices items. Perceptual and Motor Skills 36 (1973) 987-993
88. Wright D.E., and Dennis I. Exploiting the speed-accuracy trade-off. In: Ackerman P.L., Kyllonen P.C., and Roberts R.D. (Eds). Learning and individual differences: Process, trait, and content determinants (1999), American Psychological Association, Washington, DC 231-248