Comment: Selection bias, vote counting, and Money-Priming effects: A comment on rohrer, pashler, and harris (2015) and vohs (2015)

Journal of Experimental Psychology: General

Volumn 145, Issue 5, 2016, Pages 655-663

  Vadillo, Miguel A ^a   Hardwicke, Tom E ^b   Shanks, David R ^b  

^a KING'S COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Meta Analysis; Money priming; P Hacking; Publication bias; Replications

Indexed keywords

EMOTION; FEMALE; HUMAN; MALE; MOTIVATION; POLITICS; POVERTY; PSYCHOLOGY; THINKING;

EMOTIONS; FEMALE; HUMANS; MALE; MOTIVATION; POLITICS; POVERTY; THINKING;

EID: 84964344964     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/xge0000157     Document Type: Note

References (59)

1. Bishop, D. V., & Thompson, P. A. (2015). Problems in using p-curve analysis and text-mining to detect rate of p-hacking. PeerJ PrePrints, 3: e1956.
2. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. http://dx.doi.org/10.1038/nrn3475
3. Caruso, E. M., Vohs, K. D., Baxter, B., & Waytz, A. (2013). Mere exposure to money increases endorsement of free-market systems and social inequality. Journal of Experimental Psychology: General, 142, 301-306. http://dx.doi.org/10.1037/a0029288
4. Chatterjee, P., Rose, R. L., & Sinha, J. (2013). Why money meanings matter in decisions to donate time and money. Marketing Letters, 24, 109-118.
5. Copas, J. (1999). What works? Selectivity models and meta-analysis. Journal of the Royal Statistical Society: Series A: Statistics in Society, 162, 95-109. http://dx.doi.org/10.1111/1467-985X.00123
6. Dear, K. B., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237-245. http://dx.doi.org/10.1214/ss/1177011363
7. Dolan, P., Hallsworth, M., Halpern, D., King, D., & Vlaev, I. (2010). Mindspace: Influencing behaviour through public policy. London, United Kingdom: Cabinet Office.
8. Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS ONE, 7: e29081. http://dx.doi.org/10.1371/journal.pone.0029081
9. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629
10. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. http://dx.doi.org/10.1007/s11192-011-0494-7
11. Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151-156. http://dx.doi.org/10.3758/s13423-012-0227-9
12. Franco, A., Malhotra, N., & Simonovits, G. (2014, September 19). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502-1505. http://dx.doi.org/10.1126/science.1255484
13. Gasiorowska, A., Zaleskiewicz, T., & Wygrab, S. (2012). Would you do something for me? The effects of money activation on social preferences and social behavior in young children. Journal of Economic Psychology, 33, 603-608. http://dx.doi.org/10.1016/j.joep.2011.11.007
14. Gomes, C. M., & McCullough, M. E. (2015). The effects of implicit religious primes on dictator game allocations: A preregistered replication experiment. Journal of Experimental Psychology: General, 144, e94-e104. http://dx.doi.org/10.1037/xge0000027
15. Guan, M., & Vandekerckhove, J. (2016). Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review, 23, 74-86.
16. Hedges, L. V. (1992). Modeling publication selection effects in metaanalysis. Statistical Science, 7, 246-255. http://dx.doi.org/10.1214/ss/1177011364
17. Hedges, L. V., & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological Bulletin, 88, 359-369. http://dx.doi.org/10.1037/0033-2909.88.2.359
18. Hedges, L. V., & Vevea, J. (2005). Selection method approaches. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias and meta-analysis: Prevention, assessment and adjustments (pp. 145-174). New York, NY: Wiley.
19. Ioannidis, J. P. A. (2005). Differentiating biases from genuine heterogeneity: Distinguishing artifactual from substantive effects. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias and meta-analysis: Prevention, assessment and adjustments (pp. 287-302). New York, NY: Wiley. http://dx.doi.org/10.1002/0470870168.ch15
20. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253. http://dx.doi.org/10.1177/1740774507079441
21. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/0956797611430953
22. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi.org/10.1207/s15327957pspr0203_4
23. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, S., Bernstein, M. J., . . . Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45, 142-152. http://dx.doi.org/10.1027/1864-9335/a000178
24. Lakatos, I. (1978). The methodology of scientific research programmes: Philosophical papers (Vol.1). New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO9781139171434.009
25. Lakens, D. (2015). What p-hacking really looks like: A comment on Masicampo and LaLande (2012). Quarterly Journal of Experimental Psychology, 68, 829-832. http://dx.doi.org/10.1080/17470218.2014.982664
26. Lakens, D., Hilgard, J., & Staaks, J. (in press). On the reproducibility of meta-analyses: Six practical recommendations. BMC Psychology.
27. Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141. http://dx.doi.org/10.1207/s15327965pli0102_1
28. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and metaanalyses: The PRISMA statement. PLoS Medicine, 6: e1000097. http://dx.doi.org/10.1371/journal.pmed.1000097
29. Mukherjee, S., Manjaly, J. A., Kumar, N., & Shah, M. (2015). For your eyes only? Consequences of monetary primes on problem solving. Unpublished manuscript, Indian Institute of Technology Gandhinagar, Ahmedabad, India.
30. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. http://dx.doi.org/10.1177/1745691612459058
31. Open Science Collaboration. (2015, August 28). Estimating the reproducibility of psychological science. Science, 349(6251): aac4716. http://dx.doi.org/10.1126/science.aac4716
32. Park, J. K., Gasiorowska, A., & Vohs, K. D. (2015). Self-affirmation has the power to offset the harmful effects of money reminders. Unpublished manuscript, University of Delaware, Newark.
33. Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PLoS ONE, 7: e42510. http://dx.doi.org/10.1371/journal.pone.0042510
34. Pashler, H., Rohrer, D., Abramson, I., Wolfson, T., & Harris, C. (2016). A social priming data set with troubling oddities. Basic and Applied Social Psychology, 38, 3-19.
35. Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (in press). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research.
36. Rohrer, D., Pashler, H., & Harris, C. R. (2015). Do subtle reminders of money change people's political views? Journal of Experimental Psychology: General, 144, e73-e85. http://dx.doi.org/10.1037/xge0000058
37. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. http://dx.doi.org/10.1037/0033-2909.86.3.638
38. Rufibach, K. (2011). Selection models with monotone weight functions in meta analysis. Biometrical Journal, 53, 689-704. http://dx.doi.org/10.1002/bimj.201000240
39. Sarial-Abi, G., Hamilton, R., & Vohs, K. D. (2015). The motivating effect of money. Unpublished manuscript, Bocconi University, Milan, Italy.
40. Sarial-Abi, G., & Vohs, K. D. (2015). Money primes and goal pursuit. Unpublished manuscript, Bocconi University, Milan, Italy.
41. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551-566. http://dx.doi.org/10.1037/a0029487
42. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316. http://dx.doi.org/10.1037/0033-2909.105.2.309
43. Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., . . . Moore, C. (2013). Priming intelligent behavior: An elusive phenomenon. PLoS ONE, 8: e56515. http://dx.doi.org/10.1371/journal.pone.0056515
44. Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., . . . Puhlmann, L. M. C. (2015). Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives? Journal of Experimental Psychology: General, 144, e142-e158. http://dx.doi.org/10.1037/xge0000116
45. Shariff, A. F., & Norenzayan, A. (2007). God is watching you: Priming God concepts increases prosocial behavior in an anonymous economic game. Psychological Science, 18, 803-809. http://dx.doi.org/10.1111/j.1467-9280.2007.01983.x.
46. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
47. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547. http://dx.doi.org/10.1037/a0033242
48. Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Better p-curves: Making p-curve analysis more robust to errors, fraud, and ambitious p-hacking, a reply to Ulrich and Miller (2015). Journal of Experimental Psychology: General, 144, 1146-1152. http://dx.doi.org/10.1037/xge0000104
49. Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119-1129. http://dx.doi.org/10.1016/S0895-4356(00)00242-0
50. Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., . . . Higgins, J. P. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 343: d4002. http://dx.doi.org/10.1136/bmj.d4002
51. Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000). Modelling publication bias in meta-analysis: A review. Statistical Methods in Medical Research, 9, 421-445. http://dx.doi.org/10.1191/096228000701555244
52. Thaler, R., & Sunstein, C. (2008). Nudge: Improving decisions about health, wealth and happiness. New Haven, CT: Yale University Press.
53. Vadillo, M. A., Konstantinidis, E., & Shanks, D. R. (2016). Underpowered samples, false negatives, and unconscious learning. Psychonomic Bulletin & Review, 23, 87-102.
54. van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293-309. http://dx.doi.org/10.1037/met0000025.
55. van Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E.-J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6: 1365. http://dx.doi.org/10.3389/fpsyg.2015.01365
56. Vohs, K. D. (2015). Money priming can change people's thoughts, feelings, motivations, and behaviors: An update on 10 years of experiments. Journal of Experimental Psychology: General, 144, e86-e93. http://dx.doi.org/10.1037/xge0000091
57. Vohs, K. D., Mead, N. L., & Goode, M. R. (2006, November 17). The psychological consequences of money. Science, 314, 1154-1156. http://dx.doi.org/10.1126/science.1132491
58. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. http://dx.doi.org/10.1177/1745691612463078
59. Yong, E. (2012). Nobel laureate challenges psychologists to clean up their act: Social-priming research needs "daisy chain" of replication. http://dx.doi.org/10.1038/nature.2012.11535