Assessing model mimicry using the parametric bootstrap

Journal of Mathematical Psychology

Volumn 48, Issue 1, 2004, Pages 28-50

  Wagenmakers, Eric Jan ^a,d   Ratcliff, Roger ^a   Gomez, Pablo ^b   Iverson, Geoffrey J ^c  

^a NORTHWESTERN UNIVERSITY   (United States)

^b DEPAUL UNIVERSITY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 1542615630     PISSN: 00222496     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jmp.2003.11.004     Document Type: Article

References (113)

1. Aitchison, J., & Dunsmore, I. R. (1975). Statistical prediction analysis. Cambridge: Cambridge University Press.
2. Aitkin, M. (1991). Posterior Bayes factors. Journal of the Royal Statistical Society B, 53, 111-142.
3. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov, & F. Caski (Eds.), Proceeding of the second international symposium on information theory. Budapest: Akademiai Kiado.
4. Anderson, N. H. (1981). Foundations of information integration theory. New York: Academic Press.
5. Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. London: Lawrence-Erlbaum Associates.
6. Andrews, D. W. K., & Buchinsky, M. (2000). A three-step method for choosing the number of bootstrap repetitions. Econometrica, 68, 23-51.
7. Atkinson, R. C., Bower, G. H., & Crothers, E. J. (1965). An introduction to mathematical learning theory. New York: Wiley.
8. Berger, J. O. (1985). Statistical decision theory and Bayesian analysis. New York: Springer.
9. Berkhof, J., van Mechelen, I., & Gelman, A. A Bayesian approach to the selection and testing of latent class models. Statistica Sinica, in press.
10. Bollback, J. P. (2002). Bayesian model adequacy and choice in phylogenetics. Molecular Biology and Evolution, 19, 1171-1180.
11. Bollen, K. A., & Stine, R. (1992). Bootstrapping goodness of fit measures in structural equation models. Sociological Methods and Research, 21, 205-229.
12. Box, G. E. P. (1980). Sampling and Bayes' inference in scientific modelling and robustness. Journal of the Royal Statistical Society A, 143, 383-430.
13. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach. New York: Springer.
14. Bush, R. R., & Mosteller, F. (1955). Stochastic models for learning. New York: Wiley.
15. Collyer, C. E. (1985). Comparing strong and weak models by fitting them to computer-generated data. Perception and Psychophysics, 38, 476-481.
16. Corbett, A. T. (1977). Retrieval dynamics for rote and visual image mnemonics. Journal of Verbal Learning and Verbal Behavior, 16, 233-246.
17. Cox, D. R. (1962). Further results on tests of separate families of hypotheses. Journal of the Royal Statistical Society B, 24, 406-424.
18. Crowther, C. S., Batchelder, W. H., & Hu, X. (1995). A measurement-theoretic analysis of the fuzzy logical model of perception. Psychological Review, 102, 396-408.
19. Cutting, J. E. (2000). Accuracy, scope, and flexibility of models. Journal of Mathematical Psychology, 44, 3-19.
20. Cutting, J. E., Bruno, N., Brady, N. P., & Moore, C. (1992). Selectivity, scope, and simplicity of models: A lesson from fitting judgments of perceived depth. Journal of Experimental Psychology: General, 121, 364-381.
21. Davidson, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge: Cambridge University Press.
22. Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19, 55-68.
23. Diciccio, T. J., & Romano, J. P. (1988). A review of bootstrap confidence intervals. Journal of the Royal Statistical Society B, 50, 338-354.
24. Djurić, P. M. (1998). Asymptotic MAP criteria for model selection. IEEE Transactions on Signal Processing, 46, 2726-2735.
25. Dosher, B. A. (1981). The effects of delay and interference: A speed-accuracy study. Cognitive Psychology, 13, 551-582.
26. Dunn, J. C. (2000). Model complexity: The fit to random data reconsidered. Psychological Research, 63, 174-182.
27. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242.
28. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1-26.
29. Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1, 54-77.
30. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
31. Estes, W. K. (1956). The problem of inference from curves based on group data. Psychological Bulletin, 53, 134-140.
32. Estes, W. K. (2002). Traps in the route to models of memory and decision. Psychonomic Bulletin & Review, 9, 3-25.
33. Feng, Z. D., & McCulloch, C. E. (1996). Using bootstrap likelihood ratios in finite mixture models. Journal of the Royal Statistical Society B, 58, 609-617.
34. Gelman, A., Goegebeur, Y., Tuerlinckx, F., & van Mechelen, I. (2000). Diagnostic checks for discrete-data regression models using posterior predictive simulations. Applied Statistics, 49, 247-268.
35. Geweke, J. (1999a). Simulation methods for model criticism and robustness analysis. In J. M. Bernardo, J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 6 (pp. 275-299). Oxford: Oxford University Press.
36. Geweke, J. (1999b). Using simulation methods for Bayesian econometric models: Inference, development, and communication. Econometric Reviews, 18, 1-126.
37. Geweke, J., & McCausland, W. (2001). Bayesian specification analysis in econometrics. The American Journal of Agricultural Economics, 83, 1181-1186.
38. Golden, R. M. (1995). Making correct statistical inferences using a wrong probability model. Journal of Mathematical Psychology, 39, 3-20.
39. Golden, R. M. (2003). Discrepancy risk model selection test theory for comparing possibly misspecified or nonnested models. Psychometrika, 68, 229-249.
40. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.
41. Grünwald, P. (2000). Model selection based on minimum description length. Journal of Mathematical Psychology, 44, 133-152.
42. Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals. Annals of Statistics, 16, 927-953.
43. Hall, P. (1992). The bootstrap and Edgeworth expansion. New York: Springer.
44. Hall, P., & Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics, 47, 757-762.
45. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning: Data mining, inference, and prediction. New York: Springer.
46. Hinkley, D. V. (1988). Bootstrap methods. Journal of the Royal Statistical Society B, 50, 321-337.
47. Horowitz, J. L. (2001). The bootstrap. In J. J. Heckman, & E. E. Leamer (Eds.), Handbook of econometrics: Vol. 5 (pp. 3159-3228). Amsterdam: Elsevier Science.
48. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795.
49. Kass, R. E., & Wasserman, L. (1996). The selection of prior distributions by formal rules. Journal of the American Statistical Association, 91, 1343-1369.
50. Laud, P. W., & Ibrahim, J. G. (1995). Predictive model selection. Journal of the Royal Statistical Society B, 57, 247-262.
51. Massaro, D. W. (1988). Some criticisms of connectionist models of human performance. Journal of Memory and Language, 27, 213-234.
52. Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
53. Massaro, D. W., Cohen, M. M., Campbell, C. S., & Rodriguez, T. (2001). Bayes factor of model selection validates FLMP. Psychonomic Bulletin & Review, 8, 1-17.
54. Massaro, D. W., & Friedman, D. (1990). Models of integration given multiple sources of information. Psychological Review, 97, 225-252.
55. McClelland, J. L., & Elman, J. L. (1986). The TRACE model of speech perception. Cognitive Psychology, 18, 1-86.
56. McElree, B., & Carrasco, M. (1999). The temporal dynamics of visual search: Speed-accuracy tradeoff analysis of feature and conjunctive searches. Journal of Experimental Psychology: Human Perception and Performance, 25, 1517-1539.
57. McElree, B., & Dosher, B. A. (1989). Serial position and set size in short-term memory: The time course of recognition. Journal of Experimental Psychology: General, 118, 346-373.
58. McElree, B., & Dosher, B. A. (1993). Serial retrieval processes in the recovery of order information. Journal of Experimental Psychology: General, 122, 291-315.
59. McLachlan, G. J. (1987). On bootstrapping the likelihood ratio test statistic for the number of components in a normal mixture. Applied Statistics, 36, 318-324.
60. Meng, X.-L. (1994). Posterior predictive p-values. The Annals of Statistics, 22, 1142-1160.
61. Metropolis, N., & Ulam, S. (1949). The Monte Carlo method. Journal of the American Statistical Association, 44, 335-341.
62. Movellan, J. R., & McClelland, J. L. (2001). The Morton-Massaro law of information integration: Implications for models of perception. Psychological Review, 108, 113-148.
63. Myung, I. J. (2002). Comparing models with 'landscaping'. Presentation for the 1st Annual Summer Interdisciplinary Conference, Squamish (B.C.), Canada, August 2002.
64. Myung, I. J., Forster, M. R., & Browne, M. W. (Eds.), (2000). Model selection [Special issue]. Journal of Mathematical Psychology, 44(1-2).
65. Myung, I. J., & Pitt, M. A. (1997). Applying Occam's razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin and Review, 4, 79-95.
66. Navarro, D. J., Myung, I. J., Pitt, M. A., & Kim, W. (2003). Global model analysis by landscaping. Proceedings of the 25th annual conference of the cognitive science society.
67. Navarro, D. J., Pitt, M. A., & Myung, I. J. (2003). Assessing the distinguishability of models and the informativeness of data. Manuscript submitted for publication.
68. Oden, G. C., & Massaro, D. W. (1978). Integration of featural information in speech perception. Psychological Review, 85, 172-191.
69. Parzen, E., Tanabe, K., & Kitagawa, G. (Eds.), (1998). Selected papers of Hirotugu Akaike. New York: Springer.
70. Pitt, M. A., Kim, W., & Myung, I. J. (2003). Flexibility versus generalizability in model selection. Psychonomic Bulletin and Review, 10, 29-44.
71. Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472-491.
72. Press, W. H., Flannery, B. P., Teukolsky, S. A., & Vetterling, W. T. (1986). Numerical recipes. Cambridge: Cambridge University Press.
73. Raftery, A. E. (1995). Bayesian model selection in social research (with discussion). In P. V. Marsden (Ed.), Sociological methodology (pp. 111-196). Cambridge: Blackwells.
74. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59-108.
75. Ratcliff, R. (1979). Group reaction time distributions and an analysis of distribution statistics. Psychological Bulletin, 86, 446-461.
76. Ratcliff, R. (1988a). A note on mimicking additive reaction time models. Journal of Mathematical Psychology, 32, 192-204.
77. Ratcliff, R. (1988b). Continuous versus discrete information processing: Modeling accumulation of partial information. Psychological Review, 95, 238-255.
78. Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114, 510-532.
79. Ratcliff, R., Gomez, P., & McKoon, G. A diffusion model account of the lexical decision task. Psychological Review, in press.
80. Ratcliff, R. & Iverson, G. J. (1984). Methods for investigating parameters spaces of models. Paper presented at the Seventeenth Annual Mathematical Psychology Meeting, Chicago, IL.
81. Ratcliff, R., & Rouder, J. N. (1998). Modeling response times for two-choice decisions. Psychological Science, 9, 347-356.
82. Ratcliff, R., & Rouder, J. N. (2000). A diffusion model account of masking in two-choice letter identification. Journal of Experimental Psychology: Human Perception and Performance, 26, 127-140.
83. Ratcliff, R. & Smith, P. L. A comparison of sequential sampling models for two-choice reaction time. Psychological Review, in press.
84. Ratcliff, R., Thapar, A., & McKoon, G. (2001). The effects of aging on reaction time in a signal detection task. Psychology and Aging, 16, 323-341.
85. Ratcliff, R., Thapar, A., & McKoon, G. (2003). A diffusion model analysis of the effects of aging on brightness discrimination. Perception and Psychophysics, 65, 523-535.
86. Ratcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability. Psychonomic Bulletin and Review, 9, 438-481.
87. Reed, A. V. (1976). List length and the time-course of recognition in immediate memory. Memory & Cognition, 4, 16-30.
88. Rissanen, J. (1996). Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 42, 40-47.
89. Rissanen, J. (2001). Strong optimality of the normalized ML models as universal codes and information in data. IEEE Transactions on Information Theory, 47, 1712-1717.
90. Rubin, D. B. (1981). The Bayesian bootstrap. Annals of Statistics, 9, 130-134.
91. Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12, 1151-1172.
92. Rubin, D. B., & Stern, H. S. (1994). Testing in latent class models using a posterior predictive check distribution. In A. von Eye, & C. Clogg (Eds.), Latent variable analysis: Applications for Developmental Research. Thousand Oaks, CA: Sage.
93. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
94. Silverman, B. W. (1986). Density estimation for statistics and data analysis. London: Chapman & Hall.
95. Stuart, A. & Ord, J. K. (1991). Kendall's advanced theory of statistics. (Vol. 2). New York: Oxford University Press.
96. Thapar, A., Ratcliff, R., & McKoon, G. (2003). A diffusion model analysis of the effects of aging on letter discrimination. Psychology and Aging, 18, 415-429.
97. Thurman, W. N. (2001). Bayesian specification analysis in econometrics: Comment. The American Journal of Agricultural Economics, 83, 1187-1189.
98. Townsend, J. T. (1972). Some results concerning the identifiability of parallel and serial processes. British Journal of Mathematical and Statistical Psychology, 25, 168-197.
99. Usher, M., & McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review, 108, 550-592.
100. Van Zandt, T. (2002). Analysis of response time distributions. In J. T. Wixted (Ed.), Stevens' handbook of experimental psychology (pp. 461-516) Vol. 4 (3rd ed.). New York: Wiley Press.
101. Van Zandt, T., & Ratcliff, R. (1995). Statistical mimicking of reaction time data: Single-process models, parameter variability, and mixtures. Psychonomic Bulletin and Review, 2, 20-54.
102. von Neumann, J. (1951). Various techniques used in connection with random digits. National Bureau of Standards, Applied Mathematics Series, 12, 36-38.
103. Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307-333.
104. Wagenmakers, E.-J., & Farrell, S. AIC model selection using Akaike weights. Psychonomic Bulletin & Review, in press.
105. Wagenmakers, E.-J., Farrell, S., & Ratcliff, R., Naïve nonparametric bootstrap model weights are biased. Biometrics, in press.
106. Wasserman, L. (2000). Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44, 92-107.
107. Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: II. Bootstrap-based confidence intervals and sampling. Perception and Psychophysics, 63, 1314-1329.
108. Wickelgren, W. A. (1976). Network strength theory of storage and retrieval dynamics. Psychological Review, 83, 466-478.
109. Wickelgren, W. A. (1977). Speed-accuracy tradeoff and information processing dynamics. Acta Psychologica, 41, 67-85.
110. Wickelgren, W. A., Corbett, A. T., & Dosher, B. A. (1980). Priming and retrieval from short-term memory: A speed accuracy trade-off analysis. Journal of Verbal Learning and Verbal Behavior, 19, 387-404.
111. Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. Annals of Mathematical Statistics, 9, 60-62.
112. Williams, D. A. (1970). In discussion of "A method for discriminating between models" by A.C. Atkinson. Journal of the Royal Statistical Society B, 32, 350.
113. Williams, D. A. (1970a). Discrimination between regression models to determine the pattern of enzyme synthesis in synchronous cell cultures. Biometrics, 26, 23-32.