Opportunistic biases: Their origins, effects, and an integrated solution

American Psychologist

Volumn 70, Issue 6, 2015, Pages 499-514

  DeCoster, Jamie ^a   Sparks, Erin A ^b   Sparks, Jordan C ^c   Sparks, Glenn G ^b   Sparks, Cheri W ^d  

^a UNIVERSITY OF VIRGINIA   (United States)

^b PURDUE UNIVERSITY   (United States)

^c UNIVERSITY OF MINNESOTA   (United States)

^d Indiana First Steps ^*  (United States)

Author keywords

Bias; Exploratory and confirmatory analyses; Methodology; Opportunistic; Replicability; Statistics

Indexed keywords

HUMAN; PSYCHOLOGY; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS;

DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOLOGY; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84941061994     PISSN: 0003066X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0039191     Document Type: Article

References (72)

1. Abdi, H. (2007). Bonferroni and Šidák corrections for multiple comparisons. In N. J. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 103-107). Thousand Oaks, CA: Sage.
2. American Psychological Association (APA). (2010). Publication manual of the American Psychological Association, sixth edition. Washington, DC: Author.
3. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851. http://dx.doi.org/10.1037/0003-066X.63.9 .839
4. Association for Psychological Science. (2014). Registered replication reports. Retrieved March 2014 from http://www.psychologicalscience .org/index.php/replication
5. Barnett, V., & Lewis, T. (1994). Outliers in statistical data (3rd ed.). Chichester, England: Wiley.
6. Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531-533. http://dx.doi.org/ 10.1038/483531a
7. Bem, D. J. (2003). Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger III, (Eds.), The compleat academic (pp. 171-201). Washington, DC: American Psychological Association.
8. Betz, M. A., & Gabriel, K. R. (1978). Type IV errors and analysis of simple effects. Journal of Educational Statistics, 3, 121-143. http://dx .doi.org/10.2307/1164881
9. Botella, J., Ximénez, C., Revuelta, J., & Suero, M. (2006). Optimization of sample size in controlled experiments: The CLAST rule. Behavior Research Methods, 38, 65-76. http://dx.doi.org/10.3758/BF03192751
10. Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. http://dx.doi.org/10.1038/nrn3475
11. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145-153. http://dx.doi.org/10.1037/h0045186
12. Cohen, J. (1994). The Earth is round (p <. 05). American Psychologist, 49, 997-1003. http://dx.doi.org/10.1037/0003-066X.49.12.997
13. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.
14. Congdon, P. (2006). Bayesian statistical modeling. Chichester, UK: Wiley. http://dx.doi.org/10.1002/9780470035948
15. Cooper, H. (2010). Reporting research in psychology: How to meet journal article reporting standards. Washington, DC: American Psychological Association.
16. Cooper, H., & Vandenbos, G. R. (2013). Archives of scientific psychology: A new journal for a new era. Archives of Scientific Psychology, 1, 1-6. http://dx.doi.org/10.1037/arc0000001
17. Costa, P. T., & McCrae, R. R. (1992). The NEO-PIR professional manual. Odessa, FL: Psychological Assessment Resources.
18. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29. http://dx.doi.org/10.1177/0956797613504966
19. Devine, P. G. (2012, August 19). Responsible conduct in research. E-mail to SPSP members.
20. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290. http://dx.doi .org/10.1177/1745691611406920
21. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3-6. http://dx.doi.org/10.1177/0956797613512465
22. Fanelli, D. (2010). "Positive" results increase down the hierarchy of the sciences. PLoS ONE, 5,e10068.
23. Fidler, F. (2002). The fifth ed. of the APA Publication Manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62,749-770.
24. Frick, R. W. (1998). A better stopping rule for conventional statistical tests. Behavior Research Methods, Instruments, & Computers, 30, 690-697. http://dx.doi.org/10.3758/BF03209488
25. Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246-263. http://dx.doi.org/10.2307/2841583
26. Goodman, S. N. (1993). p Values, hypothesis tests, and likelihood: Implications for epidemiology of a neglected historical debate. American Journal of Epidemiology, 137, 485-496.
27. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20. http://dx.doi.org/10.1037/ h0076157
28. Hays, W. L. (1963). Statistics for psychologists. New York, NY: Holt, Rinehart & Winston.
29. Hubbard, R., & Armstrong, J. S. (1997). Publication bias against null results. Psychological Reports, 80, 337-338. http://dx.doi.org/10.2466/ pr0.1997.80.1.337
30. Hubbard, R., & Ryan, P. A. (2000). The historical growth of statistical significance testing in psychology-and its future prospects. Educational and Psychological Measurement, 60,661-681.
31. Ioannidis, J. P. A. (2005). Why most published research findings are false.PLoSMedicine, 2, e124. http://dx.doi.org/10.1371/journal.pmed .0020124
32. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. http://dx.doi .org/10.1177/1740774507079441
33. Jha, A. (2012). Tenfold increase in scientific research papers retracted for fraud. The Guardian. Retrieved October 2012 from http://www.guardian .co.uk/science/2012/oct/01/tenfold-increase-science-paper-retractedfraud
34. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/ 0956797611430953
35. Johnson, V. E. (2013). Revised standards for statistical evidence. PNAS Proceedings of the National Academy of Sciences of the United States of America, 110, 19313-19317. http://dx.doi.org/ 10.1073/pnas.1313476110
36. Journal for Experimental Social Psychology. (2014). JESP editorial guidelines. Retrieved May 2014 from http://www.journals.elsevier.com/journalof-experimental-social-psychology/news/jesp-editorial-guidelines
37. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi .org/10.1207/s15327957pspr0203_4
38. Knott, D., Muers, S., & Aldridge, S. (2008). Achieving culture change: A policy framework. London, UK: Prime Minister's Strategy Unit. Retrieved October 2012 from http://webarchive.nationalarchives.gov.uk/ 20100125070726/http://cabinetoffice.gov.uk/media/cabinetoffice/strategy/ assets/achieving_culture_change.pdf
39. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299-312. http://dx.doi.org/10.1177/1745691611406925
40. Kupfersmid, J. (1988). Improving what is published: A model in search of an ed. American Psychologist, 43, 635-642. http://dx.doi.org/10.1037/ 0003-066X.43.8.635
41. Laws, K. R. (2013). Negativland: A home for all findings in psychology. BMC Psychology, 1, 2. http://dx.doi.org/10.1186/2050-7283-1-2
42. Leamer, E. E. (1978). Specification searches: Ad hoc inference with non-experimental data. New York, NY: Wiley.
43. LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424-432. http://dx.doi.org/ 10.1177/1745691613491437
44. Light, R., & Pillemer, D. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
45. MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490-504. http://dx.doi .org/10.1037/0033-2909.111.3.490
46. Many Labs Replication Group. (2014). Investigating variation in replicability: The "Many Labs" Replication Project. Retrieved March 2014 from https://osf.io/wx7ck/
47. Morey, L. C. (1991). Personality Assessment Inventory professional manual. Odessa, FL: Psychological Assessment Resources.
48. Natrella, M. G. (1960). The relation between confidence intervals and tests of significance. The American Statistician, 14, 20-22, 38.
49. Neher, A. (1967). Probability pyramiding, research error and the need for independent replication. The Psychological Record, 17, 257-262.
50. Pashler, H., & Wagenmakers, E. J. (Eds.) (2012). Introduction to the special section on replicability in psychological science: A crisis of confidence. Perspectives on Psychological Science, 7, 528-530. http:// dx.doi.org/10.1177/1745691612465253
51. Pierce, J., Nosek, B. A., Miguez, S., Holcombe, A. O., Spies, J. R., de-Wit, L., & Lewis, M. (2013). Standards (and badges) for encouraging open science behaviors. Retrieved July 2013 from http://open scienceframework.org/project/TVyXZ/
52. Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika, 64, 191-199. http://dx.doi.org/10.1093/ biomet/64.2.191
53. Prelec, D. (2004). A Bayesian truth serum for subjective data. Science, 306, 462-466. http://dx.doi.org/10.1126/science.1102081
54. Reproducibility Project & the Psychology Group. (2014). Reproducibility Project: Psychology. Retrieved March 2014 from http://openscience framework.org/project/EZcUj/wiki/home
55. Robinson, D. H., & Wainer, H. (2002). On the past and future of null hypothesis significance testing. The Journal of Wildlife Management, 66, 263-271. http://dx.doi.org/10.2307/3803158
56. Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641. http://dx.doi.org/10.1037/ 0033-2909.86.3.638
57. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551-566. http://dx.doi.org/10.1037/a0029487
58. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90-100. http://dx.doi.org/10.1037/a0015108
59. Selvin, H. C., & Stewart, A. (1966). Data-dredging procedures in survey analysis. The American Statistician, 20, 20-23.
60. Shaver, J. P., & Norton, R. S. (1980). Randomness and replication in ten years of the American Educational Research Journal. Educational Researcher, 9, 9-15.
61. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
62. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012). A 21 word solution. http://dx.doi.org/10.2139/ssrn.2160588
63. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597-599. http://dx .doi.org/10.1177/1745691612463399
64. Simonsohn, U. (2014). Posterior-hacking: Selective reporting invalidates Bayesian results also. http://dx.doi.org/10.2139/ssrn.2374040
65. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547. http://dx.doi.org/10.1037/a0033242
66. Spies, J., & Nosek, B. (2014). Open science framework. Retrieved March 2014 from http://openscienceframework.org/
67. Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., . . . Schinke, S. P. (2011). Replication in prevention science. Prevention Science, 12, 103-117. http://dx.doi.org/10.1007/s11121-011-0217-6
68. van Assen, M. A., van Aert, R. C., Nuijten, M. B., & Wicherts, J. M. (2014). Why publishing everything is more effective than selective publishing of statistically significant results. PLoS ONE, 9, e84896. http://dx.doi.org/10.1371/journal.pone.0084896
69. Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. http://dx.doi.org/ 10.1177/1745691612463078
70. Wald, A. (1947). Sequential analysis. New York, NY: Wiley.
71. Walster, G. W., & Cleary, T. A. (1970). A proposal for a new editorial policy in the social sciences. The American Statistician, 24, 5-10.
72. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E. J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298. http://dx.doi.org/10.1177/1745691611406923