Practicing what we preach (and sometimes study): Methodological issues in experimental laboratory research

Review of General Psychology

Volumn 19, Issue 2, 2015, Pages 191-202

  Nichols, Austin Lee ^a   Edlund, John E ^b  

^a UNIVERSITY OF NAVARRA   (Spain)

^b ROCHESTER INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Experiment; Experimenter; Laboratory; Methodology; Research

Indexed keywords

EID: 84933177836     PISSN: 10892680     EISSN: 19391552     Source Type: Journal    
DOI: 10.1037/gpr0000027     Document Type: Article

References (80)

1. Adair, J. G. (1973). The human subject: The social psychology of the psychological experiment. Boston, MA: Little, Brown.
2. Adair, J. G., & Epstein, J. S. (1968). Verbal cues in the mediation of experimenter bias. Psychological Reports, 22, 1045-1053. http://dx.doi .org/10.2466/pr0.1968.22.3c.1045
3. Anderson, C. A., & Bushman, B. J. (1997). External validity of "trivial" experiments: The case of laboratory aggression. Review of General Psychology, 1, 19-41. http://dx.doi.org/10.1037/1089-2680.1.1.19
4. Azrin, N. H., Holz, W., Ulrich, R., & Goldiamond, I. (1961). The control of the content of conversation through reinforcement. Journal of the Experimental Analysis of Behavior, 4, 25-30. http://dx.doi.org/10.1901/ jeab.1961.4-25
5. Blackhart, G. C., Brown, K. E., Clark, T., Pierce, D. L., & Shell, K. (2012). Assessing the adequacy of postexperimental inquiries in deception research and the factors that promote participant honesty. Behavior Research Methods, 44, 24 - 40. http://dx.doi.org/10.3758/s13428-011-0132-6
6. Carey, B. (2011, November 2). Fraud case seen as a red flag for psychology research. The New York Times. Retrieved from http://www.nytimes .com/2011/11/03/health/research/noted-dutch-psychologist-stapelaccused- of-research-fraud.html
7. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112-130. http://dx .doi.org/10.3758/s13428-013-0365-7
8. Crespi, L. P. (1946). The cheater problem in polling. Public Opinion Quarterly, 9, 431-445. http://dx.doi.org/10.1086/265760
9. DeSantis, A., & Kayson, W. A. (1997). Defendants' characteristics of attractiveness, race, and sex and sentencing decisions. Psychological Reports, 81, 679-683. http://dx.doi.org/10.2466/pr0.1997.81.2.679
10. DeYoung, C. G., Peterson, J. B., & Higgins, D. M. (2002). Higher-order factors of the Big Five predict conformity: Are there neuroses of health? Personality and Individual Differences, 33, 533-552. http://dx.doi.org/ 10.1016/S0191-8869(01)00171-4
11. Diener, E., Matthews, R., & Smith, R. E. (1972). Leakage of experimental information to potential future subjects by debriefed subjects. Journal of Experimental Research in Personality, 6, 264-267.
12. Dion, K., Berscheid, E., & Walster, E. (1972). What is beautiful is good. Journal of Personality and Social Psychology, 24, 285-290. http://dx .doi.org/10.1037/h0033731
13. Durgin, F. H., Baird, J. A., Greenburg, M., Russell, R., Shaughnessy, K., & Waymouth, S. (2009). Who is being deceived? The experimental demands of wearing a backpack. Psychonomic Bulletin & Review, 16, 964-969. http://dx.doi.org/10.3758/PBR.16.5.964
14. Edlund, J. E., Hartnett, J. H., Heider, J. H., Perez, E. J., & Lusk, J. (2014). Experimenter characteristics and word choice: Best practices when administering an informed consent. Ethics & Behavior, 24, 397-407. http://dx.doi.org/10.1080/10508422.2013.854171
15. Edlund, J. E., Nichols, A. L., Okdie, B. M., Guadagno, R. E., Eno, C. A., Heider, J. D.,... Wilcox, K. T. (2014). The prevalence and prevention of crosstalk: A multi-institutional study. The Journal of Social Psychology, 154, 181-185. http://dx.doi.org/10.1080/00224545.2013.872596
16. Edlund, J. E., Sagarin, B. J., & Johnson, B. S. (2007). Reciprocity and the belief in a just world. Personality and Individual Differences, 43, 589-596. http://dx.doi.org/10.1016/j.paid.2007.01.007
17. Edlund, J. E., Sagarin, B. J., Skowronski, J. J., Johnson, S. J., & Kutter, J. (2009). Whatever happens in the laboratory stays in the laboratory: The prevalence and prevention of participant crosstalk. Personality and Social Psychology Bulletin, 35, 635-642. http://dx.doi.org/10.1177/ 0146167208331255
18. Farrow, J. M., Farrow, B. J., Lohss, W. E., & Taub, S. I. (1975). Intersubject communication as a contaminating factor in verbal conditioning. Perceptual and Motor Skills, 40, 975-982. http://dx.doi.org/10.2466/ pms.1975.40.3.975
19. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science, 7, 555-561. http://dx.doi.org/ 10.1177/1745691612459059
20. Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571. http://dx.doi.org/ 10.1177/1745691612457576
21. Glinski, R. J., Glinski, B. C., & Slatin, G. T. (1970). Nonnaivety contamination in conformity experiments: Sources, effects, and implications for control. Journal of Personality and Social Psychology, 16, 478-485. http://dx.doi.org/10.1037/h0030073
22. Golding, S. L., & Lichtenstein, E. (1970). Confession of awareness and prior knowledge of deception as a function of interview set and approval motivation. Journal of Personality and Social Psychology, 14, 213-223. http://dx.doi.org/10.1037/h0028853
23. Gross, L. R. (1959). Effects of verbal and nonverbal reinforcement in the Rorschach. Journal of Consulting Psychology, 23, 66-68. http://dx.doi .org/10.1037/h0043654
24. Guest, L. (1947). A study of interviewer competence. International Journal of Opinion & Attitude Research, 1, 17-30.
25. Hakmiller, K. L. (1971). Inter-subject communication. Psychological Reports, 28, 956-958. http://dx.doi.org/10.2466/pr0.1971.28.3.956
26. Hansel, C. E. M. (1966). ESP: A scientific evaluation. New York, NY:Charles Scribner's Sons.
27. Haslam, S. A., & Reicher, S. D. (2012). Contesting the "nature" of conformity: What Milgram and Zimbardo's studies really show. PLoS Biology, 10, e1001426. http://dx.doi.org/10.1371/journal.pbio.1001426
28. Haslam, S. A., Reicher, S. D., & Birney, M. (2014). Nothing by mere authority: Evidence that in an experimental analogue of the Milgram paradigm participants are motivated not by orders but by appeals to science. Journal of Social Issues, 70, 473- 488. http://dx.doi.org/ 10.1111/josi.12072
29. Haslam, S. A., Reicher, S. D., Millard, K., & McDonald, R. (2014). "Happy to have been of service": The Yale archive as a window into the engaged followership of participants in Milgram's 'obedience' experiments. British Journal of Social Psychology. Advance online publication. http://dx.doi.org/10.1111/bjso.12074
30. Highhouse, S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554 -566. http://dx.doi.org/10.1177/ 1094428107300396
31. Horka, S. T., & Farrow, B. J. (1970). A methodological note on intersubject communication as a contaminating factor in psychological experiments. Journal of Experimental Child Psychology, 10, 363-366. http:// dx.doi.org/10.1016/0022-0965(70)90058-5
32. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399-425. http://dx.doi.org/10.1007/s10683-011-9273-9
33. Hyman, H. H. (1954). Interviewing in social research. Chicago, IL: University of Chicago Press.
34. Jung, J. (1982). The experimenter's challenge: Methods and issues in psychological research. New York, NY: Macmillan.
35. Kintz, B. L., Delprato, D. J., Mettee, D. R., Persons, C. E., & Schappe, R. H. (1965). The experimenter effect. Psychological Bulletin, 63, 223-232. http://dx.doi.org/10.1037/h0021718
36. Klein, O., Doyen, S., Leys, C., Magalhaes de Saldanha da Gama, P. A., Miller, S., Questienne, L., & Cleeremans, A. (2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7, 572-584. http:// dx.doi.org/10.1177/1745691612463704
37. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J.,... Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45, 142-152. http://dx.doi.org/10.1027/1864-9335/a000178
38. Langlois, J. H., Kalakanis, L., Rubenstein, A. J., Larson, A., Hallam, M., & Smoot, M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychological Bulletin, 126, 390-423.
39. Levin, S. M. (1961). The effects of awareness on verbal conditioning. Journal of Experimental Psychology, 61, 67-75. http://dx.doi.org/ 10.1037/h0047321
40. Levy, L. H. (1967). Awareness, learning, and the beneficent subject as expert witness. Journal of Personality and Social Psychology, 6, 365-370. http://dx.doi.org/10.1037/h0024716
41. Lichtenstein, E. (1970). "Please don't talk to anyone about this experiment":Disclosure of deception by debriefed subjects. Psychological Reports, 26, 485-486. http://dx.doi.org/10.2466/pr0.1970.26.2.485
42. Logue, A. W. (1998). Laboratory research on self-control: Applications to administration. Review of General Psychology, 2, 221-238. http://dx.doi .org/10.1037/1089-2680.2.2.221
43. Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537-542. http://dx.doi.org/10.1177/ 1745691612460688
44. Marwit, S. J., & Marcia, J. E. (1967). Tester bias and response to projective instruments. Journal of Consulting Psychology, 31, 253-258. http://dx .doi.org/10.1037/h0024662
45. Mills, J. (1976). A procedure for explaining experiments involving deception. Personality and Social Psychology Bulletin, 2, 3-13. http://dx.doi .org/10.1177/014616727600200102
46. Mills, T. M. (1962). A sleeper variable in small groups research: The experimenter. Pacific Sociological Review, 5, 21-28. http://dx.doi.org/ 10.2307/1388274
47. Minor, M. W. (1970). Experimenter-expectancy effect as a function of evaluation apprehension. Journal of Personality and Social Psychology, 15, 326-332. http://dx.doi.org/10.1037/h0029612
48. Morton, T. A., Haslam, S. A., Postmes, T., & Ryan, M. K. (2006). We value what values us: The appeal of identity-affirming science. Political Psychology, 27, 823-838. http://dx.doi.org/10.1111/j.1467-9221.2006 .00539.x
49. Newberry, B. H. (1973). Truth telling in subjects with information about experiments: Who is being deceived? Journal of Personality and Social Psychology, 25, 369-374. http://dx.doi.org/10.1037/h0034229
50. Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151-166. http://dx.doi.org/10.3200/GENP.135.2.151-166
51. Orne, M. T. (1962). On the social psychology of the psychological experiment:With particular reference to demand characteristics and their implications. American Psychologist, 17, 776-783. http://dx.doi.org/ 10.1037/h0043424
52. Orne, M. T. (1969). Demand characteristics and the concept of quasicontrols. In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in behavioral research. New York, NY: Academic Press.
53. Page, M. (1973). On detecting demand awareness by post-experimental questionnaire. The Journal of Social Psychology, 91, 305-323. http://dx .doi.org/10.1080/00224545.1973.9923053
54. Reicher, S. D., Haslam, S. A., & Smith, J. R. (2012). Working towards the experimenter: Reconceptualizing obedience within the Milgram paradigm as identification-based followership. Perspectives on Psychological Science, 7, 315-324. http://dx.doi.org/10.1177/1745691612448482
55. Roediger, H. L. (2012). Psychology's woes and a partial cure: The value of replication. Observer, 25, 27-29.
56. Rosenthal, R. (1963). On the social psychology of the psychological experiment: The experimenter's hypothesis as unintended determinant of experimental results. American Scientist, 51, 268-283.
57. Rosenthal, R. (1964). Experimental outcome-orientation and the results of the psychological experiment. Psychological Bulletin, 61, 405-412. http://dx.doi.org/10.1037/h0045850
58. Rosenthal, R. (1966). Experimenter effects in behavioral research. New York, NY: Appleton-Century-Crofts.
59. Rosenthal, R. (1968). Experimenter expectancy and the reassuring nature of the null hypothesis decision procedure. Psychological Bulletin, 70, 30-47. http://dx.doi.org/10.1037/h0026727
60. Rosenthal, R., & Fode, K. L. (1963). Psychology of the scientist: V. Three experiments in experimenter bias. Psychological Reports, 12, 491-511. http://dx.doi.org/10.2466/pr0.1963.12.2.491
61. Rosenthal, R., Persinger, G. W., Vikan-Kline, L., & Fode, K. L. (1963). The effect of experimenter outcome bias and subject set on awareness in verbal conditioning experiments. Journal of Verbal Learning and Verbal Behavior, 2, 275-283. http://dx.doi.org/10.1016/S0022-5371(63)80094-8
62. Rosenthal, R., & Rosnow, R. L. (1975). The volunteer subject. New York, NY: Wiley.
63. Sagarin, B. J., Rhoads, K., & Cialdini, R. B. (1998). Deceiver's distrust:Denigration as a consequence of undiscovered deception. Personality and Social Psychology Bulletin, 24, 1167-1176. http://dx.doi.org/ 10.1177/01461672982411004
64. Schultz, D. P. (1969). The human subject in psychological research. Psychological Bulletin, 72, 214 -228. http://dx.doi.org/10.1037/ h0027880
65. Seemann, E. A., Carroll, S. J., Woodard, A., & Mueller, M. L. (2008). The type of threat matters: Differences in similar magnitude threats elicit differing magnitudes of psychological reactance. North American Journal of Psychology, 10, 583-603.
66. Shames, M. L., & Adair, J. G. (1980). Effect of experimenter's expectancy in relation to type and structure of the experimental task. Psychological Reports, 46, 487-497. http://dx.doi.org/10.2466/pr0.1980.46.2.487
67. Sigall, H., Aronson, H., & Van Hoose, T. (1970). The cooperative subject:Myth or reality? Journal of Experimental Social Psychology, 6, 1-10. http://dx.doi.org/10.1016/0022-1031(70)90072-7
68. Silverman, I. (1968). The effects of experimenter outcome expectancy on latency of word association. Journal of Clinical Psychology, 24, 60-63. http://dx.doi.org/10.1002/1097-4679(196801)24:1<60::AIDJCLP2270240115>3.0.CO;2-4
69. Silverman, I. (1977). The human subject in the psychological laboratory. New York, NY: Pergamon Press.
70. Smith, R. B., III, Helm, B., & Tedeschi, J. T. (1972). Verbal conditioning of "aware" subjects. The Journal of Social Psychology, 87, 151-152. http://dx.doi.org/10.1080/00224545.1972.9918662
71. Spielberger, C. D., & Levin, S. M. (1962). What is learned in verbal conditioning? Journal of Verbal Learning & Verbal Behavior, 1, 125-132. http://dx.doi.org/10.1016/S0022-5371(62)80009-7
72. Stricker, L. J., Messick, S., & Jackson, D. N. (1967). Suspicion of deception:Implications for conformity research. Journal of Personality and Social Psychology, 5, 379-389. http://dx.doi.org/10.1037/h0021209
73. Taub, S. I., & Farrow, B. J. (1973). Reinforcement effects on intersubject communication: The scuttlebutt effect. Perceptual and Motor Skills, 37, 15-22. http://dx.doi.org/10.2466/pms.1973.37.1.15
74. Taylor, K., & Shepperd, J. (1996). Probing suspicion among participants in deception research. American Psychologist, 51, 886-887. http://dx.doi .org/10.1037/0003-066X.51.8.886
75. Walsh, W. B. (1976). Disclosure of deception by debriefed subjects:Another look. Psychological Reports, 38, 783-786. http://dx.doi.org/ 10.2466/pr0.1976.38.3.783
76. Walsh, W. B., & Stillman, S. M. (1974). Disclosure of deception by debriefed subjects. Journal of Counseling Psychology, 21, 315-319. http://dx.doi.org/10.1037/h0036683
77. Weber, S. J., & Cook, T. D. (1972). Subject effects in laboratory research:An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77, 273-295. http://dx.doi.org/ 10.1037/h0032351
78. Wellman, J. A., & Geers, A. L. (2009). Rebel without a (conscious) cause:Priming a nonconscious goal for psychological reactance. Basic and Applied Social Psychology, 31, 9 -16. http://dx.doi.org/10.1080/ 01973530802659711
79. White, H. A., & Schumsky, D. A. (1972). Prior information and "awareness" in verbal conditioning. Journal of Personality and Social Psychology, 24, 162-165. http://dx.doi.org/10.1037/h0033308
80. Wickes, T. A., Jr. (1956). Examiner influence in a testing situation. Journal of Consulting Psychology, 20, 23-26. http://dx.doi.org/10.1037/ h0048983