On the limitations of manipulation checks: An obstacle toward cumulative science

International Review of Social Psychology

Volumn 30, Issue 1, 2017, Pages 125-130

  Fayant, Marie Pierre ^a   Sigall, Harold ^b   Lemonnier, Aurore ^a   Retsin, Emilie ^a   Alexopoulos, Theodore ^a  

^a UNIVERSITÉ PARIS DESCARTES   (France)

^b UNIVERSITY OF MARYLAND   (United States)

Author keywords

Best research practices; Construct validity; Experimentation; Manipulation check

Indexed keywords

EID: 85039424708     PISSN: None     EISSN: 23978570     Source Type: Journal    
DOI: 10.5334/irsp.102     Document Type: Article

References (26)

1. Bless, H., & Burger, A. M. (2016). A closer look at social psychologists’ silver bullet: Inevitable and evitable side effects of the experimental approach. Perspectives On Psychological Science, 11, 296–308. DOI: https://doi.org/10.1177/1745691615621278
2. Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., van’t Veer, A., et al. (2014). The Replication Recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. DOI: https://doi.org/10.1016/j.jesp.2013.10.005
3. Brewer, M. B., & Crano, W. D. (2014). Research design and issues of validity. In: Reis, H. T., & Judd, C. M. (Eds.), Handbook of Research Methods in Social and Personality Psychology, 2nd ed. (pp. 11–26). New York, NY, US: Cambridge University Press.
4. Chaiken, S. (1980). Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of Personality and Social Psychology, 39, 752–766. DOI: https://doi.org/10.1037/0022-3514.39.5.752
5. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312. DOI: https://doi.org/10.1037/0003-066X.45.12.1304
6. Cook, T. D., Campbell, D. T., & Peracchio, L. (1990). Quasi Experimentation. In: Dunnette, M. D., & Hough, L. M. (Eds.), Handbook of Industrial and Organizational Psychology. Palo Alto, CA: Consulting Psychologists Press.
7. Deutsch, M., & Krauss, R. M. (1965). Theories in Social Psychology. NY: Basic Books.
8. Fiedler, K., Schott, M., & Meiser, T. (2011). What mediation analysis can (not) do. Journal of Experimental Social Psychology, 47, 1231–1236. DOI: https://doi.org/10.1016/j.jesp.2011.05.007
9. Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015). Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108, 275–297. DOI: https://doi.org/10.1037/pspi0000007
10. Flake, J. K., Pek, J., & Hehman, E. (2017). Construct Validation in Social and Personality Research: Current Practice and Recommendations. Social Psychological and Personality Science. DOI: https://doi.org/10.1177/1948550617693063
11. Foschi, M. (2014). Hypotheses, operationalizations, and manipulation checks. In: Webster, M. J., & Sell, J. (Eds.), Laboratory Experiments in the Social Sciences, 2nd ed. (pp. 247–268). San Diego, CA, US: Elsevier Academic Press. DOI: https://doi.org/10.1016/B978-0-12-404681-8.00011-X
12. Haslam, S. A., & McGarty, C. (2003). Experimental design and causality in social psychological research. In: Sansone, C., Morf, C. C., & Panter, A. T. (Eds.), Handbook of Methods in Social Psychology, (pp. 237–264), Thousand Oaks, CA: Sage.
13. Hüffmeier, J., Mazei, J., & Schultze, T. (2016). Recon-ceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology, 66, 81–92. DOI: https:// doi.org/10.1016/j.jesp.2015.09.009
14. Jacoby, J., & Sassenberg, K. (2011). Interactions do not only tell us when, but can also tell us how: Testing process hypotheses by interaction. European Journal of Social Psychology, 41. DOI: https://doi.org/10.1002/ejsp.762
15. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. DOI: https://doi.org/10.1177/0956797611430953
16. Kühnen, U. (2010). Manipulation checks as manipulation: Another look at the ease-of-retrieval heuristic. Personality and Social Psychology Bulletin, 36, 47–58. DOI: https://doi.org/10.1177/0146167209346746
17. Maxwell, S. E. (2004). The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies. Psychological Methods, 9, 147–163. DOI: https://doi.org/10.1037/1082-989X.9.2.147
18. Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14, 5–8. DOI: https://doi.org/10.1111/j.1745-3992.1995.tb00881.x
19. O’Keefe, D. J. (2003). Message Properties, Mediating States, and Manipulation Checks: Claims,
20. Evidence, and Data Analysis in Experimental Persuasive Message Effects Research. Communication Theory, 13, 251–274. DOI: https://doi.org/10.1111/j.1468-2885.2003.tb00292.x
21. Platt, J. R. (1964). Strong inference. Science, 146, 347–353. DOI: https://doi.org/10.1126/science.146.3642.347
22. Reis, H. T., & Judd, C. M. (2014). Handbook of Research Methods in Social and Personality Psychology, 2nd ed. New York: Cambridge University Press.
23. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA, US: Houghton, Mifflin and Company.
24. Sigall, H., & Mills, J. (1998). Measures of independent variables and mediators are useful in social psychology experiments: But are they necessary? Personality and Social Psychology Review, 2, 218–226. DOI: https://doi.org/10.1207/s15327957pspr0203_5
25. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71. DOI: https://doi.org/10.1177/1745691613514450
26. Wilson, T. D., Aronson, E., & Carlsmith, K. (2010). The art of laboratory experimentation. In: Fiske, S. T., Gilbert, D. T., & Lindzey, G. (Eds.), Handbook of Social Psychology, 1, 51–81. NJ: Wiley. DOI: https://doi.org/10.1002/9780470561119. socpsy001002