Using Nonnaive Participants Can Reduce Effect Sizes

Psychological Science

Volumn 26, Issue 7, 2015, Pages 1131-1139

  Chandler, Jesse ^a,b   Paolacci, Gabriele ^c   Peer, Eyal ^d   Mueller, Pam ^e   Ratliff, Kate A ^f  

^a UNIVERSITY OF MICHIGAN   (United States)

^b MATHEMATICA POLICY RESEARCH   (United States)

^c ERASMUS UNIVERSITY   (Netherlands)

^d BAR ILAN UNIVERSITY   (Israel)

^e PRINCETON UNIVERSITY   (United States)

^f UNIVERSITY OF FLORIDA   (United States)

Author keywords

effect sizes; judgment and decision making; nonna vet ; open data; open materials; panel conditioning; repeated participation; research methods

Indexed keywords

ADULT; FEMALE; HUMAN; MALE; PATIENT PARTICIPATION; PATIENT SELECTION; PROCEDURES;

ADULT; FEMALE; HUMANS; MALE; PATIENT PARTICIPATION; PATIENT SELECTION;

EID: 84936990843     PISSN: 09567976     EISSN: 14679280     Source Type: Journal    
DOI: 10.1177/0956797615585115     Document Type: Article

References (31)

1. M.H.Birnbaum (1999). How to show that 9 > 221: Collect judgments in a between-subjects design. Psychological Methods, 4, 243–249.
2. J.ChandlerP.MuellerG.Paolacci (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.
3. G.CharnessU.GneezyM.A.Kuhn (2012). Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81, 1–8.
4. F.FaulE.ErdfelderA.BuchnerA.-G.Lang (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160. doi:10.3758/BRM.41.4.1149
5. C.FoxA.Tversky (1995). Ambiguity aversion and comparative ignorance. The Quarterly Journal of Economics, 110, 585–603.
6. J.Goyder (1986). Surveys on surveys: Limits and potentialities. Public Opinion Quarterly, 50, 27–41. doi:10.1086/268957
7. A.G.Greenwald (1976). Within-subjects designs: To use or not to use?Psychological Bulletin, 83, 314–320.
8. A.G.GreenwaldD.E.McGheeJ.L.K.Schwartz (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480.
9. A.G.GreenwaldB.A.Nosek (2001). Health of the Implicit Association Test at age 3. Zeitschrift für Experimentelle Psychologie, 48(2), 85–93.
10. J.C.HersheyP.J.H.Schoemaker (1980). Prospect theory’s reflection hypothesis: A critical examination. Organizational Behavior and Human Performance, 25, 395–418.
11. D.S.HillygusN.JacksonM.Young (2014). Professional respondents in nonprobability online panels. In M.CallegaroR.BakerJ.BethlehemA.S.GöritzJ.A.KrosnickP.J.Lavrakas (Eds.), Online panel research: A data quality perspective (pp. 219–237). West Sussex, England: Wiley.
12. E.R.HirtS.J.Sherman (1985). The role of prior knowledge in explaining hypothetical events. Journal of Experimental Social Psychology, 21, 519–543.
13. S.HusnuR.J.Crisp (2010). Elaboration enhances the imagined contact effect. Journal of Experimental Social Psychology, 46, 943–950.
14. H.H.HymanP.B.Sheatsley (1950). The current status of American public opinion. In J.C.Payne (Ed.), The teaching of contemporary affairs (pp. 11–34). New York, NY: National Council for the Social Studies.
15. K.E.JacowitzD.Kahneman (1995). Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21, 1161–1166.
16. M.K.JohnsonS.HashtroudiD.S.Lindsay (1993). Source monitoring. Psychological Bulletin, 114, 3–28.
17. R.A.KleinK.A.RatliffM.VianelloR.B.AdamsJr.Š.BahníkM.J.BernsteinB.A.. Nosek (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152.
18. M.W.LipseyD.B.Wilson (Eds.). (2001). Applied Social Research Methods Series: Vol. 49. Practical meta-analysis. Thousand Oaks, CA: Sage.
19. I.LorgeC.C.Curtis (1936). Prestige, suggestion, and attitudes. The Journal of Social Psychology, 7, 386–402.
20. W.MasonS.SuriD.J.Watts (2014). Long-run learning in games of cooperation. In Proceedings of the Fifteenth ACM Conference on Economics and Computation (pp. 821–838). New York, NY: Association for Computing Machinery.
21. D.M.OppenheimerT.MeyvisN.Davidenko (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
22. D.M.OppenheimerB.Monin (2009). The retrospective gambler’s fallacy: Unlikely events, constructing the past, and multiple universes. Judgment and Decision Making, 4, 326–334.
23. G.PaolacciJ.Chandler (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.
24. D.G.RandA.PeysakhovichG.T.Kraft-ToddG.E.NewmanO.WurzbacherM.A.NowakJ.D.Greene (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, Article 3677. Retrieved from http://www.nature.com/ncomms/2014/140422/ncomms4677/full/ncomms4677.html
25. F.D.RichardC.F.BondJr.J.J.Stokes-Zoota (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.
26. D.Rugg (1941). Experiments in wording questions: II. Public Opinion Quarterly, 5, 91–92.
27. N.SchwarzH.J.HipplerB.DeutschF.Strack (1985). Response scales: Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly, 49, 388–395.
28. S.J.Sherman (1980). On the self-erasing nature of errors of prediction. Journal of Personality and Social Psychology, 39, 211–221.
29. P.SturgisN.AllumI.Brunton-Smith (2009). Attitudes over time: The psychology of panel conditioning. In P.Lynn (Ed.), Methodology of Longitudinal Surveys (pp. 113–126). West Sussex, England: John Wiley & Sons.
30. A.TverskyD.Kahneman (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458.
31. S.J.WeberT.D.Cook (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77, 273–295.