Small is beautiful: In defense of the small-N design

Psychonomic Bulletin and Review

Volumn 25, Issue 6, 2018, Pages 2083-2101

  Smith, Philip L ^a   Little, Daniel R ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Inference; Mathematical psychology; Methodology; Replication

Indexed keywords

COGNITION; GROUP PROCESS; HUMAN; INDIVIDUALITY; LEAST SQUARE ANALYSIS; MATHEMATICS; METHODOLOGY; PSYCHOLOGY; PSYCHOPHYSICS; REACTION TIME; SAMPLE SIZE; VISION;

COGNITION; GROUP PROCESSES; HUMANS; INDIVIDUALITY; LEAST-SQUARES ANALYSIS; MATHEMATICS; PSYCHOLOGY; PSYCHOPHYSICS; REACTION TIME; RESEARCH DESIGN; SAMPLE SIZE; VISUAL PERCEPTION;

EID: 85044183247     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-018-1451-8     Document Type: Article

References (130)

1. Aarts, A., Anderson, J., Anderson, C., Attridge, P., & Attwood, A. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 1–8
2. Ashby, F. G. (1982). Deriving exact predictions from the cascade model. Psychological Review, 89, 599–607
3. Ashby, F. G., & Gott, R. E. (1988). Decision rules in the perception and categorization of multidimensional stimuli. Journal of Experimental Psychology: Learning, Memory and Cognition, 14, 33–53
4. Ashby, F. G., & Lee, W. W. (1991). Predicting similarity and categorization from identification. Journal of Experimental Psychology: General, 120, 150–172
5. Ashby, F. G., & Alfonso-Reese, L.A. (1995). Categorization as probability density estimation. Journal of Mathematical Psychology, 39, 216–233
6. Baddeley, A.D. (1997) Human memory: Theory and practice. East Sussex: Psychology Press
7. Baker, M. (2015). First results from psychology’s largest reproducibility test. Retrieved from. 10.1038/nature.2015.17433
8. Baribault, B., Donkin, C., Little, D.R., Trueblood, J.S., Orzvecz, Z., van Ravenzwaaij, D., & vanderkerckhove, J. (2018). Robust tests of theory with randomly sampled experiments. Proceedings of the National Academy of Sciences [in press]
9. Bartlema, A., Lee, M., Wetzels, R., & Vanpaemel, W. (2014). A Bayesian hierarchical mixture approach to individual differences: Case studies in selective attention and representation in category learning. Journal of Mathematical Psychology, 59, 132–150
10. Batchelder, W. H. (1975). Individual differences and the all-or-none vs incremental learning controversy. Journal of Mathematical Psychology, 12, 53–74
11. Benjamin, D.J., Berger, J., Johannesson, M., Nosek, B.A., Wagenmakers, E.-J., Berk, R., & et al. (2017). Redefine statistical significance. Nature Human Behaviour. Retrieved from. 10.1038/s41562-017-0189-z
12. Boring, E. G. (1954). The nature and history of experimental control. The American Journal of Psychology, 67, 573–589
13. Brown, S. D., & Heathcote, A (2008). The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive Psychology, 57, 153–178
14. Bruner, J. S., & Goodman, C. C (1947). Value and need as organizing factors in perception. The Journal of Abnormal and Social Psychology, 42, 33–44
15. Bundesen, C. (1990). A theory of visual attention. Psychological Review, 97, 523–547
16. Busemeyer, J. R. (1985). Decision making under uncertainty: A comparison of simple scalability, fixedsample, and sequential-sampling models. Journal of Experimental Psychology: Learning Memory and Cognition, 11, 538–564
17. Busemeyer, J. R., & Townsend, J. T (1993). Decision field theory: A dynamic-cognitive approach to decision making in an uncertain environment. Psychological Review, 100, 432–459
18. Carter, L.F., & Schooler, K. (1949). Value, need, and other factors in perception. Psychological Review, 56, 200–207
19. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312
20. Coppola, D. M., White, L. E., Fitzpatrick, D., & Purves, D. (1998). Unequal representation of cardinal and oblique contours in ferret visual cortex. Proceedings of the National Academy of Sciences, 95, 2621–2623
21. Cowan, N. (1999). An embedded-processes model of working memory. Models of working memory: Mechanisms of active maintenance and executive control, 20, 62–101
22. Cumming, G., & Finch, S (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170–180
23. de Lange, H.D. (1952). Experiments on flicker and some calculations on an electrical analogue of the foveal systems. Physica, 18, 935–950
24. de Lange, D.H. (1954). Relationship between critical flicker-frequency and a set of low-frequency characteristics of the eye. JOSA, 44, 380–389
25. de Lange, D.H. (1958). Research into the dynamic nature of the human fovea-cortex systems with intermittent and modulated light. I. attenuation characteristics with white and colored light. JOSA, 777–784
26. Dennis, S., & Humphreys, M. S (2001). A context noise model of episodic word recognition. Psychological Review, 108, 452–478
27. Dorfman, D. D., & Biderman, M (1971). A learning model for a continuum of sensory states. Journal of Mathematical Psychology, 8, 264–284
28. Dosher, B. A., & Lu, Z.-L. (1999). Mechanisms of perceptual learning. Vision Research, 39, 3197–3221
29. Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS one, 7, e29081
30. Estes, W. K. (1956). The problem of inference from curves based on group data. Psychological Bulletin, 53, 134–140
31. Estes, W. K. (1997). On the communication of information by displays of standard errors and confidence intervals. Psychonomic Bulletin & Review, 4, 330–341
32. Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences about cognitive processes from model fits to individual versus average performance. Psychonomic Bulletin & Review, 12, 403–408
33. Fifić, M., Little, D.R., & Nosofsky, R. (2010). Logical-rule models of classification response times: A synthesis of mental-architecture, random-walk, and decision-bound approaches. Psychological Review, 117, 309–348
34. Firestone, C., & Scholl, B. J. (2016). Cognition does not affect perception: Evaluating the evidence for top-down effects. Behavioral and Brain Sciences, 39, 1–77
35. Fisher, R.A. (1925) Statistical methods for research workers. Guildford: Genesis Publishing Pvt Ltd
36. Furmanski, C. S., & Engel, S. A. (2000). An oblique effect in human primary visual cortex. Nature Neuroscience, 3, 535–536
37. Gallistel, C. R., Fairhurst, S., & Balsam, P. (2004). The learning curve: Implications of a quantitative analysis. Proceedings of the National Academy of Sciences of the United States of America, 101, 13124–13131
38. Gelman, A. (2015). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management, 41, 632–643
39. Gelman, A., & Loken, E. (2014). The statistical crisis in science data-dependent analysis—a garden of forking paths—explains why many statistically significant comparisons don’t hold up. American Scientist, 102, 460–465
40. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) Bayesian data analysis. London: Chapman and Hall CRC
41. Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on Estimating the reproducibility of psychological science. Science, 351, 1037–1037
42. Graham, N.V.S. (1989) Visual pattern analyzers. Oxford: Oxford University Press
43. Grice, J., Barrett, P., Cota, L., Felix, C., Taylor, Z., Garner, S., & et al. (2017). Four bad habits of modern psychologists. Behavioral Sciences, 7, 53–74
44. Heath, R. A. (1992). A general nonstationary diffusion model for two-choice decision-making. Mathematical Social Sciences, 23, 283–309
45. Henrich, J., Heine, S. J., & Noranyazan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61–135
46. Houpt, J. W., & Townsend, J. T. (2010). The statistical properties of the survivor interaction contrast. Journal of Mathematical Psychology, 54, 446–453
47. Houpt, J. W., & Townsend, J. T. (2012). Statistical measures for workload capacity analysis. Journal of Mathematical Psychology, 56, 341–355
48. Jones, M., Curran, T., Mozer, M. C., & Wilder, M. H. (2013). Sequential effects in response time reveal learning mechanisms and event representations. Psychological Review, 120, 628–666
49. Kac, M. (1962). A note on learning signal detection. IRE Transactions on Information Theory, 8, 126–128
50. Kahneman, D. (2011) Thinking, fast and slow. Basingstoke: Macmillan
51. Keppel, G. (1982) Design and analysis: A researcher’s handbook. Upper Saddle River: Prentice-Hall
52. Kerlinger, F. N., & Lee, H. B. (1999) Foundations of behavioral research. Belmont: Wadsworth Publishing
53. Kruschke, J. (2014) Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan. Cambridge: Academic Press
54. Lakens, D., Adolfi, F.G., Albers, C.J., Anvari, F., Apps, M.A.J., Argamon, S. E., & et al. (2017). Justify your alpha: A response to redefine statistical significance. PsyArXiv, 1-18. Retrieved from. 10.17605/OSF.IO/9S3Y6
55. ′≡ 0: A dynamic model for binary prediction. Journal of Mathematical Psychology, 60(35), 46
56. Lee, M. D., & Wagenmakers, E.-J. (2005). Bayesian statistical inference in psychology: Comment on Trafimow. Psychological Review, 112, 662–668
57. Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605–621
58. Liang, J., & Bentler, P. M. (2004). An EM algorithm for fitting two-level structural equation models. Psychometrika, 69, 101–122
59. Liew, S. X., Howe, P. D. L., & Little, D. R. (2016). The appropriacy of averaging in the study of context effects. Psychonomic Bulletin & Review, 23, 1639–1646
60. Link, S.W. (1992) The wave theory of difference and similarity. Hillsdale: Erlbaum
61. Little, D. R., & Smith, P. L. (2018). Commentary on Zwaan et al. – replication is already mainstream: Lessons from small-N designs. Behavioral and Brain Sciences [in press]
62. Little, D. R., Wang, T., & Nosofsky, R. M. (2016). Sequence-sensitive exemplar and decision-bound accounts of speeded-classification performance in a modified Garner-tasks paradigm. Cognitive Psychology, 89, 1–38
63. Little, D. R., Altieri, N., Fifić, M., & Yang, C.-T. (2017) Systems factorial technology: A theory driven methodology for the identification of perceptual and cognitive mechanisms. New York: Academic Press
64. Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476–490
65. Lu, Z.-L., & Dosher, B. A. (1999). Characterizing human perceptual inefficiencies with equivalent internal noise. JOSA A, 16, 764–778
66. Luce, R. D. (1986) Response times: Their role in inferring elementary mental organization. New York: Oxford University Press
67. Maxwell, S.E., & Delaney, H. D. (1990) Designing experiments and analyzing data: A model comparison approach. Wadsworth: Belmont
68. McClelland, J. L. (1979). On the time relations of mental processes: An examination of systems of processes in cascade. Psychological Review, 86, 287–330
69. McCullagh, P., & Nelder, J. A. (1989) Generalized linear models. Boca Raton: Chapman & Hall
70. Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115
71. Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244
72. Miller, J. (1988). Discrete and continuous models of human information processing: Theoretical distinctions and empirical results. Acta Psychologica, 67, 191–257
73. Myung, J. I., & Pitt, M. A. (2004). Model comparison methods. Methods in Enzymology, 383, 351–366
74. Navarro, D. J., Griffiths, T. L., Steyvers, M., & Lee, M. D. (2006). Modeling individual differences using Dirichlet processes. Journal of Mathematical Psychology, 50, 101–122
75. Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. Computer Journal, 7, 308–313
76. Normand, M.P. (2016). Less is more: Psychologists can learn more by studying fewer people. Frontiers in Psychology, 7, 934. Retrieved from. 10.3389/fpsyg.2016.00934
77. Nosofsky, R. M. (1984). Choice, similarity, and the context theory of classification. Journal of Experimental Psychology: Learning Memory, and Cognition, 10, 104–114
78. Nosofsky, R. M. (1986). Attention, similarity, and the identification-categorization relationship. Journal of Experimental Psychology: General, 115, 39–61
79. Nosofsky, R. M., Little, D. R., Donkin, C., & Fifić, M. (2011). Short-term memory scanning viewed as exemplar-based categorization. Psychological Review, 118, 280–315
80. Nosofsky, R. M., & Palmeri, T. (1997). An exemplar-based random walk model of speeded classification. Psychological Review, 104, 266–300
81. OSC, et al. (2015). Estimating the reproducibility of psychological science. Science, 349, 1–8
82. Osth, A. F., & Dennis, S. (2015). Sources of interference in item and associative recognition memory. Psychological Review, 122, 260–311
83. Pashler, H., Coburn, N., & Harris, C.R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PloS one, 7, e42510
84. Raaijmakers, J. G. W., & Shiffrin, R. M. (1981). Search of associative memory. Psychological Review, 88, 93–144
85. Racine, J., Su, L., & Ullah, A. (2014) The Oxford handbook of applied nonparametric and semiparametric econometrics and statistics. Oxford: Oxford University Press
86. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59–108
87. Ratcliff, R., & Childers, R. (2015). Individual differences and fitting methods for the two-choice diffusion model of decision making. Decision, 15, 237–279
88. Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: Theory and data for two-choice decision tasks. Neural Computation, 20, 873–922
89. Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111, 333–367
90. Ratcliff, R., Thapar, A., & Mckoon, G. (2003). A diffusion model analysis of the effects of aging on brightness discrimination. Attention, Perception, & Psychophysics, 65, 523–535
91. Ratcliff, R., Thapar, A., Gomez, P., & McKoon, G. (2004). A diffusion model analysis of the effects of aging in the lexical-decision task. Psychology and Aging, 19, 278
92. Robson, J. G. (1966). Spatial and temporal contrast-sensitivity functions of the visual system. JOSA, 56, 1141–1142
93. Roe, R. M., Busemeyer, J. R., & Townsend, J. T. (2001). Multialternative decision field theory: A dynamic connectionst model of decision making. Psychological Review, 108, 370–392
94. Rohrer, D., Pashler, H., & Harris, C. R. (2015). Do subtle reminders of money change people’s political views? Journal of Experimental Psychology: General, 144, e73–e85
95. Rosenthal, B. G. (1968). Attitude toward money, need, and methods of presentation as determinants of perception of coins from 6 to 10 years of age. The Journal of General Psychology, 78, 85– 103
96. Ross, H.E. (1990). Environmental influences on geometrical illusions. In Fechner Day 90: Proceeding of the 6th annual meeting of the International Society of Psychophysicists (p. 216)
97. Ross, J. (2009). Visual perception 1950–2000. Inside Psychology: A Science Over 50 Years, 243–252
98. Rouder, J. N., & Haaf, J. M. (in press). Power, dominance, and constraint: A note on the appeal of different design traditions. Advances in Methods and Practice in Psychological Science
99. Rouder, J. N., Yue, Y., Speckman, P. L., Pratte, M. S., & Province, J. M. (2010). Gradual growth versus shape invariance in perceptual decision making. Psychological Review, 117, 1267
100. Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56, 356–374
101. Sanders, A. F. (1990). Issues and trends in the debate on discrete vs. continuous processing of information. Acta Psychologica, 74, 123–167
102. Saville, B. K., & Buskist, W. (2003). Traditional idiographic approaches: Small-N research designs. In S. F. Davis, B. K. Saville, & W. Buskist (Eds.) Handbook of research methods in experimental psychology (pp. 66–82). Hoboken: Blackwell Publishing Ltd
103. Schweickert, R. (1978). A critical path generalization of the additive factor methods analysis of a Stroop task. Journal of Mathematical Psychology, 18, 105–139
104. Schweickert, R. (1980). Critical-path scheduling of mental processes in a dual task. Science, 209, 704–706
105. Schweickert, R., & Mounts, J. (1998). Additive effects of factors on reaction time and evoked potentials in continuous-flow models. In C.E. Dowling, F. S. Roberts, & P. Theuns (Eds.) Recent progress in mathematical psychology: Psychophysics, knowledge, representation, cognition, and measurement (pp. 311–327). HIllsdale, NJ: Erlbaum
106. Segall, M. H., Campbell, D. T., & Herskovits, M. J. (1963). Cultural differences in the perception of geometric illusions. Science, 139, 769–771
107. Seidenberg, M. S., & McClelland, J. L. (1989). A distributed, developmental model of word recognition and naming. Psychological Review, 96, 523
108. Sheynin, O. B. (1979). C. F. Gauss and the theory of errors. Archive for History of Exact Sciences, 20, 21–72
109. Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REM-retrieving effectively from memory. Psychonomic Bulletin & Review, 4, 145–166
110. Sidman, M. (1960). Tactics of scientific research, New York: Basic Books
111. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366
112. Skinner, B.F. (1938). The behaviour of organisms: An experimental analysis. New York: Appleton-Century
113. Smith, P. L. (1995). Psychophysically principled models of visual simple reaction time. Psychological Review, 102, 567–593
114. Smith, P. L. (1998). Attention and luminance detection: A quantitative analysis. Journal of Experimental Psychology: Human Perception and Performance, 24, 105–133
115. Smith, P. L. (2016). Diffusion theory of decision making in continuous report. Psychological Review, 123, 425–451
116. Smith, P. L., & Ratcliff, R. (2009). An integrated theory of attention and decision making in visual signal detection. Psychological Review, 116, 282–317
117. Sperling, G., & Sondhi, M. M. (1968). Model for visual luminance discrimination and flicker detection. JOSA, 58, 1133–1145
118. Sternberg, S. (1969). The discovery of processing stages: Extensions of Donders’ method. Acta Psychologica, 30, 276–315
119. Stevens, S.S. (1951) Mathematics, measurement, and psychophysics. New York: Wiley
120. Switkes, E., Mayer, M. J., & Sloan, J. A. (1978). Spatial frequency analysis of the visual environment: Anisotropy and the carpentered environment hypothesis. Vision Research, 18, 1393–1399
121. Thiele, J. E., Haaf, J. M., & Rouder, J.N. (2017). Is there variation across individuals in processing? Bayesian analysis for systems factorial technology. Journal of Mathematical Psychology, 81, 40–54
122. Thomas, E. A. C. (1973). On a class of additive learning models: Error-correcting and probability matching. Journal of Mathematical Psychology, 10, 241–264
123. Thomas, E. A. C., & Ross, B. H. (1980). On appropriate procedures for combining probability distributions within the same family. Journal of Mathematical Psychology, 21, 136–152
124. Townsend, J. T. (1990). Truth and consequences of ordinal differences in statistical distributions: Toward a theory of hierarchical inference. Psychological Bulletin, 108, 551
125. Townsend, J. T., & Nozawa, G. (1995). Spatio-temporal properties of elementary perception: An investigation of parallel, serial and coactive theories. Journal of Mathematical Psychology, 39, 321–340
126. Treisman, M., & Williams, T. C. (1984). A theory of criterion setting with an application to sequential dependencies. Psychological Review, 91, 68–111
127. Usher, M., & McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review, 108, 550–592
128. Van der Heijden, P. G. M., Dessens, J., & Bockenholt, U (1996). Estimating the concomitant-variable latent-class model with the EM algorithm. Journal of Educational and Behavioral Statistics, 21, 215–229
129. Watson, A. B. (1986). Temporal sensitivity. In K. Boff, L. Kaufman, & J. Thomas (Eds.) Handbook of perception and human performance, vol. 1 (pp. 6.1–6.43), New York: Wiley
130. Woodworth, R. S., & Schlosberg, H. (1954) Experimental psychology. New York: Holt