Survey Satisficing Inflates Reliability and Validity Measures: An Experimental Comparison of College and Amazon Mechanical Turk Samples

Educational and Psychological Measurement

Volumn 76, Issue 6, 2016, Pages 912-932

  Hamby, Tyler ^a   Taylor, Wyn ^a  

^a UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

MTurk; psychometrics; satisficing; survey design

Indexed keywords

EID: 84992567978     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164415627349     Document Type: Article

References (40)

1. Barge S., Gehlbach H., (2012). Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education, 53, 182-200. doi:10.1007/s11162-011-9251-2
2. Bass B. M., Cascio W. F., O’Conner E. J., (1974). Magnitude estimation of expressions of frequency and amount. Journal of Applied Psychology, 59, 313-320. doi:10.1037/h0036653
3. Buhrmester M., Kwang T., Gosling S. D., (2011). Amazon’s mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3-5. doi:10.1177/1745691610393980
4. Cacioppo J. T., Petty R. E., Kao C. F., (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306-307. doi:10.1207/s15327752jpa4803_13
5. Campbell D. T., Fiske D. W., (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
6. Casler K., Bickel L., Hackett E., (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 21-56. doi:10.1016/j.chb.2013.05.009
7. Cerasoli C. P., Nicklin J. M., Ford M. T., (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis. Psychological Bulletin, 140, 980-1008. doi:10.1037/a0035661
8. Crump M. J. C., McDonnell J. V., Gureckis T. M., (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS One, 8(3), e57410. doi:10.1371/journal.pone.0057410
9. Diener E., Emmons R. A., Larsen R. J., Griffin S., (1985). The Satisfaction with Life Scale. Journal of Personality Assessment, 49, 71-75. doi:10.1207/s15327752jpa4901_13
10. Dobson K. S., Mothersill K. J., (1979). Equidistant category labels for construction of Likert-type scales. Perceptual and Motor Skills, 49, 575-580. doi:10.2466/pms.1979.49.2.575
11. Goodman J. K., Cryder C. E., Cheema A., (2012). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. doi:10.1002/bdm.1753
12. Hamby T., (2015). An examination of the relationships between response scale length, label format, reliability and validity (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession Order No. 3709723)
13. Hamby T., Ickes W., (2015). Do the readability and average item length of personality scales affect their reliability? Some meta-analytic answers. Journal of Individual Differences, 36, 54-63. doi:10.1027/1614-0001/a000154
14. Hamby T., Levine D., (2016). Response-scale formats and psychological distances between categories. Applied Psychological Measurement, 40, 73-75. doi:10.1177/0146621615597961
15. International Personality Item Pool. (2001). A scientific collaboratory for the development of advanced measures of personality traits and other individual differences. Retrieved from http://ipip.ori.org/
16. John O. P., Donahue E. M., Kentle R. L., (1991). The Big Five Inventory—Versions 4a and 5a. Berkeley: University of California at Berkeley, Institute of Personality and Social Research.
17. Krosnick J. A., (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236. doi:10.1002/acp.2350050305
18. Krosnick J. A., (1999). Survey research. Annual Review of Psychology, 50, 537-567. doi:10.1146/annurev.psych.50.1.537
19. Krosnick J. A., Alwin D. F., (1988). A test of the form-resistant correlation hypothesis: Ratings, rankings, and the measurement of values. Public Opinion Quarterly, 52, 526-538. doi:10.1086/269128
20. Krosnick J. A., Narayan S., Smith W. R., (1996). Satisficing in surveys: Initial evidence. New Directions for Program Evaluation, 70, 29-44. doi:10.1002/ev.1033
21. Krosnick J. A., Presser S., (2010). Question and questionnaire design. In Wright J. D., Marsden P. V., (Eds.), Handbook of survey research (pp. 263-314). San Diego, CA: Elsevier.
22. Lelkes Y., Krosnick J. A., Marx D. M., Judd C. M., Park B., (2012). Complete anonymity compromises the accuracy of self-reports. Journal of Experimental Social Psychology, 48, 1291-1299. doi:10.1016/j.jesp.2012.07.002
23. Lennox R. D., Wolfe R. N., (1984). Revision of the Self-Monitoring Scale. Journal of Personality and Social Psychology, 46, 1349-1364. doi:10.1037//0022-3514.46.6.1349
24. Malhotra N., (2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72, 914-934. doi:10.1093/poq/nfn050
25. Narayan S., Krosnick J. A., (1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly, 60, 58-88. doi:10.1086/297739
26. Nunnally J. C., Bernstein I. H., (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
27. Oppenheimer D. M., Meyvis T., Davidenko N., (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867-872. doi:10.1016/j.jesp.2009.03.009
28. Paolacci G., Chandler J., (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. doi:10.1177/0963721414531598
29. Paolacci G., Chandler J., Ipeirotis P. G., (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419. doi:10.2139/ssrn.2100631
30. Peer E., Vosgerau J., Acquisti A., (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023-1031. doi:10.3758/s13428-013-0434-y
31. Peterson R. A., (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450-461. doi:10.1086/323732
32. Peterson R. A., Kim Y., (2013). On the relationship between coefficient alpha and composite reliability. Journal of Applied Psychology, 98, 194-198. doi:10.1037/a0030767
33. Rand D. G., (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172-179. doi:10.1016/j.jtbi.2011.03.004
34. Rosenberg M., (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.
35. Rouse S. V., (2015). A reliability analysis of Mechanical Turk data. Computers in Human Behavior, 43, 304-307. doi:10.1016/j.chb.2014.11.004
36. Schwarz N., Strack F., (1985). Cognitive and affective processes in judgments of subjective well-being: A preliminary model. In Brandstatter H., Kirchler E., (Eds.), Economic Psychology (pp. 439-447). Linz, Austria: Tauner.
37. Simon H. A., (1956). Rational choice and the structure of the environment. Psychological Review, 63, 129-138. doi:10.1037/h0042769
38. Tourangeau R., Rasinski K. A., (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 3, 299-314. doi:10.1037//0033-2909.103.3.299
39. Watson D., O’Hara M. W., Kotov R., Simms L. J., Chmielewski M., McDade-Montez E. A... Stuart S., (2007). Development and validation of the Inventory of Depression and Anxiety Symptoms (IDAS). Psychological Assessment, 19, 253-268. doi:10.1037/1040-3590.19.3.253
40. Zhang C., Conrad F. G., (2013). Speeding in web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods, 8, 127-135. doi:10.18148/srm/2014.v8i2.5453#sthash.L6o1Pduf.dpuf