Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice

Review of General Psychology

Volumn 15, Issue 4, 2011, Pages 371-379

  LeBel, Etienne P ^a   Peters, Kurt R ^b  

^a UNIVERSITY OF WESTERN ONTARIO   (Canada)

^b NORWICH UNIVERSITY   (United States)

Author keywords

Close replication; File drawer problem; Modal research practice; NHST; Psi

Indexed keywords

EID: 83155173531     PISSN: 10892680     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0025172     Document Type: Article

References (63)

1. Amir Y., Sharon I. Replication research: A " must" for the scientific advancement of psychology. Replication research in the social sciences 1991, 51-69. Sage, Newbury Park, CA. J.W. Neuliep (Ed.).
2. Bakan D. The test of significance in psychological research. Psychological Bulletin 1966, 66:1-29. doi: 10.1037/h0020412.
3. Bem D.J. Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 2011, 100:407-425. doi: 10.1037/a0021524.
4. Borsboom D., Mellenbergh G.J., van Heerden J. The concept of validity. Psychological Review 2004, 111:1061-1071. doi: 10.1037/0033-295X.111.4.1061.
5. Cohen J. The earth is round (p < .05). American Psychologist 1994, 49:997-1003. doi: 10.1037/0003-066X.49.12.997.
6. Collins H.M. Discussion: When do scientists prefer to vary their experiments. Studies in the History and Philosophy of Science 1984, 15:169-174. doi: 10.1016/0039-3681(84)90005-0.
7. Cook T.D., Groom C. The methodological assumptions of social psychology: The mutual dependence of substantive theory and method choice. The Sage handbook of methods in social psychology 2004, 19-44. Sage, Thousand Oaks, CA. C. Sansone, C.C. Morf, A.T. Panter (Eds.).
8. Cumming G. Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science 2008, 3:286-300. doi: 10.1111/j.1745-6924.2008.00079.x.
9. Dawes R.M. Rational choice in an uncertain world 1988, Harcourt Brace Jovanovich, San Diego, CA.
10. Dunlap K. The experimental methods of psychology. Psychologies of 1925 1926, 331-351. Clark University Press, Worcester, MA. C. Murchison (Ed.).
11. DeShon R.P. Measures are not invariant across groups without error variance homogeneity. Psychology Science 2004, 46:137-149.
12. Duhem, P. (1954). The aim and structure of physical theory (P. P. Wiener, Trans.). Princeton, NJ: Princeton University Press.
13. Falk R. Replication-A step in the right direction. Theory & Psychology 1998, 8:313-321. doi: 10.1177/0959354398083002.
14. Falk R., Greenbaum C.W. Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology 1995, 5:75-98. doi: 10.1177/0959354395051004.
15. Fanelli D. " Positive" results increase down the hierarchy of the sciences. PLoS ONE 2010, 5:e10068. doi: 10.1371/journal.pone.0010068.
16. Fisher R.A. Statistical methods for research workers 1926, Oliver & Boyd, London, England. 1st ed.
17. Fisher R.A. Statistical methods for research workers 1934, Oliver & Boyd, London, England. 5th ed.
18. Gawronski B., Deutsch R., Banse R. Response interference tasks as indirect measures of automatic associations. Cognitive methods in social psychology 2011, 78-123. Guilford Press, New York, NY. K.C. Klauer, C. Stahl, A. Voss (Eds.).
19. Gergen K.J. Toward transformation in social knowledge 1982, Springer-Verlag, New York, NY.
20. Greenwald A.G., Gonzalez R., Harris R.J., Guthrie D. Effect size and p values: What should be reported and what should be replicated?. Psychophysiology 1996, 33:175-183. doi: 10.1111/j.1469-8986.1996.tb02121.x.
21. Guttman L. What is not what in statistics. The Statistician 1997, 26:81-107. doi: 10.2307/2987957.
22. Hendrick C. Replications, strict replications, and conceptual replications: Are they important?. Replication research in the social sciences 1991, 41-49. Sage, Newbury Park, CA. J.W. Neuliep (Ed.).
23. Kashy D.A., Donnellan M.B., Ackerman R.A., Russell D.W. Reporting and interpreting research in PSPB: Practices, principles, and pragmatics. Personality and Social Psychology Bulletin 2009, 35:1131-1142. doi: 10.1177/0146167208331253.
24. Kirk R.E. Practical significance: A concept whose time has come. Educational and Psychological Measurement 1996, 56:746-759. doi: 10.1177/0013164496056005002.
25. Kline R.B. Beyond significance testing 2004, American Psychological Association, Washington, DC, doi: 10.1037/10693-000.
26. Kruschke J.K. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science 2010, 1:658-676. doi: 10.1002/wcs.72.
27. LeBel E.P., Paunonen S.V. Sexy but often unreliable: Impact of unreliability on the replicability of experimental findings involving implicit measures. Personality and Social Psychology Bulletin 2011, 37:570-583. doi: 10.1177/0146167211400619.
28. Lehrer J. The truth wears off: Is there something wrong with the scientific method?. The New Yorker 2010, December 13, 52. Retrieved from http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer.
29. Lindsay R.M., Ehrenberg A.S.C. The design of replicated studies. The American Statistician 1993, 47:217-228. doi: 10.2307/2684982.
30. Lykken D.T. Statistical significance in psychological research. Psychological Bulletin 1968, 70:151-159. doi: 10.1037/h0026141.
31. Maxwell S.E. The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 2004, 9:147-163. doi: 10.1037/1082-989X.9.2.147.
32. McGuire W.J. The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology 1973, 26:446-456. doi: 10.1037/h0034345.
33. Meehl P.E. Theory testing in psychology and physics: A methodological paradox. Philosophy of Science 1967, 34:103-115. doi: 10.1086/288135.
34. Meehl P.E. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 1978, 46:806-834. doi: 10.1037/0022-006X.46.4.806.
35. Meehl P.E. Why summaries of research on psychological theories are often uninterpretable. Psychological Reports 1990, 66:195-244. doi: 10.2466/PR0.66.1.195-244.
36. Messick S. Validity. Educational measurement 1989, 13-103. American Council on Education, Washington, DC. 3rd ed. R.L. Linn (Ed.).
37. Michell J. Quantitative science and the definition of measurement in psychology. British Journal of Psychology 1997, 88:355-383. doi: 10.1111/j.2044-8295.1997.tb02641.x.
38. Miller J. What is the probability of replicating a statistically significant effect?. Psychonomic Bulletin & Review 2009, 16:617-640. doi: 10.3758/PBR.16.4.617.
39. Moonesinghe R., Khoury M.J., Janssens C.J.W. Most published research findings are false-But a little replication goes a long way. PLoS Medicine 2007, 4:218-221. doi: 10.1371/journal.pmed.0040028.
40. Mulaik S.A., Raju N.S., Harshman R.A. There is a time and a place for significance testing. What if there were no significance tests? 1997, 65-115. Erlbaum, Mahwah, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
41. Mulkay M., Gilbert G.N. Replication and mere replication. Philosophy of the Social Sciences 1986, 16:21-37. doi: 10.1177/004839318601600102.
42. Oppenheimer D.M., Meyvis T., Davidenko N. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 2009, 45:867-872. doi: 10.1016/j.jesp.2009.03.009.
43. Peters, K. R. (2011). Cronbach's challenge: Putting psychological explanation in context. Unpublished manuscript, The University of Western Ontario, London, Ontario, Canada.
44. Popper K.R. The logic of scientific discovery 1959, Basic Books, New York, NY.
45. Popper K.R. Conjectures and refutations 1963, Routledge and Kegan Paul, London, England.
46. Quine W.V.O. Two dogmas of empiricism. From a logical point of view 1953, 20-46. Harvard University Press, Cambridge, MA. 2nd ed.
47. Quine W.V.O., Ullian J.S. The web of belief 1978, Random House, New York, NY. 2nd ed.
48. Rosenthal R. The " file drawer" problem and tolerance for null results. Psychological Bulletin 1979, 86:638-641.
49. Rosenthal R. Replication in behavioral research. Replication research in the social sciences 1991, 1-30. Sage, Newbury Park, CA. J.W. Neuliep (Ed.).
50. Rosenthal R. Cumulating evidence. A handbook for data analysis in the behavioral sciences: Methodological issues 1993, 519-559. Erlbaum, Hillsdale, NJ.
51. Rozeboom W.W. Good science is abductive, not hypothetico-deductive. What if there were no significance tests? 1997, 335-391. Erlbaum, Hillsdale, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
52. Runkel P.J. Casting nets and testing specimens: Two grand methods of psychology 2007, Living Control Systems, Hayward, CA.
53. Sohn D. Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology 1998, 8:291-311. doi: 10.1177/0959354398083001.
54. Thompson B. Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development 1992, 70:434-438.
55. Thompson B. AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher 1996, 25:26-30.
56. Tukey J.W. Analyzing data: Sanctification or detective work?. American Psychologist 1969, 24:83-91. doi: 10.1037/h0027108.
57. Tukey J.W. The philosophy of multiple comparisons. Statistical Science 1991, 6:100-116. doi: 10.1214/ss/1177011945.
58. Vacha-Haase T., Ness C., Nilsson J., Reetz D. Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education 1999, 67:335-341. doi: 10.1080/00220979909598487.
59. Wagenmakers E.-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 2007, 14:779-804. doi: 10.3758/BF03194105.
60. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H.L.J. Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology 2011, 100:426-432. doi: 10.1037/a0022790.
61. Wallach L., Wallach M.A. Gergen versus the mainstream: Are hypotheses in social psychology subject to empirical test?. Journal of Personality and Social Psychology 1994, 67:233-242. doi: 10.1037/0022-3514.67.2.233.
62. Wilkinson L. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 1999, 54:594-604. the Task Force on Statistical Inference, doi: 10.1037/0003-066X.54.8.594.
63. Zimmer C. It's science, but not necessarily right. The New York Times 2011, June 25, Retrieved from http://www.nytimes.com/2011/06/26/opinion/sunday/26ideas.html.