Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English

Personality and Individual Differences

Volumn 75, Issue , 2015, Pages 47-52

  Feitosa, Jennifer ^a   Joseph, Dana L ^a   Newman, Daniel A ^b  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Crowdsourcing; Data collection techniques; Invariance testing; Measurement; Survey methods

Indexed keywords

EID: 84911408250     PISSN: 01918869     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.paid.2014.11.017     Document Type: Article

References (29)

1. Barger, P. B., & Sinar, E. F. (April, 2011). Psychological data from Amazon.com's MTurk: Rapid and inexpensive - but high-quality? Presented at 26th Annual SIOP Convention, Chicago, IL.
2. Behrend T.S., Sharek D.J., Meade A.W., Wiebe E.N. The viability of crowdsourcing for survey research. Behavior Research Methods 2011, 43(3):800-813.
3. Benet-Martinez V., John O.P. Los Cinco Grandes across cultures and ethnic groups: Multitrait-multimethod analyses of the Big Five in Spanish and English. Journal of Personality and Social Psychology 1998, 75(3):729.
4. Berinsky A.J., Huber G.A., Lenz G.S. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20(3), 351-368. Human Relations 2012, 61(8):1139-1160.
5. Buhrmester M., Kwang T., Gosling S.D. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science 2011, 6(1):3-5.
6. Chandler J., Mueller P., Paolacci G. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods 2014, 46(1):112-130.
7. Cheung G.W., Rensvold R.B. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling 2002, 9(2):233-255.
8. Denissen J.J., Geenen R., Van Aken M.A., Gosling S.D., Potter J. Development and validation of a Dutch translation of the Big Five Inventory (BFI). Journal of Personality Assessment 2008, 90(2):152-157.
9. Drasgow F. Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin 1984, 95(1):134-135.
10. Gosling S.D., Vazire S., Srivastava S., John O.P. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist 2004, 59(2):93-104.
11. Hopwood C.J., Donnellan M.B. How should the internal structure of personality inventories be evaluated?. Personality and Social Psychology Review 2010, 14(3):332-346.
12. Hui C.H., Triandis H.C. Measurement in cross-cultural psychology: A review and comparison of strategies. Journal of Cross-Cultural Psychology 1985, 16(2):131-152.
13. John O.P., Donahue E.M., Kentle R.L. The big five inventory - Versions 4a and 54 (technical report) 1991, Institute of Personality and Social Research, University of California, Berkeley.
14. Joseph D.L., Newman D.A. Discriminant validity of self-reported emotional intelligence: A multitrait-multisource study. Educational and Psychological Measurement 2010, 70(4):672-694.
15. Landis R.S., Beal D.J., Tesluk P.E. A comparison of approaches to forming composite measures in structural equation models. Organizational Research Methods 2000, 3(2):186-207.
16. Lyness K.S., Kropf M.B. Cultural values and potential nonresponse bias: A multilevel examination of cross-national differences in mail survey response rates. Organizational Research Methods 2007, 10(2):210-224.
17. Meade A.W., Johnson E.C., Braddy P.W. Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology 2008, 93(3):568-592.
18. Meade A.W., Lautenschlager G.J. A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods 2004, 7(4):361-388.
19. Nye C.D., Newman D.A., Joseph D.L. Never say "always"? Extreme item wording effects on scalar invariance and item response curves. Organizational Research Methods 2010, 13(4):806-830.
20. Nye C.D., Roberts B.W., Saucier G., Zhou X. Testing the measurement equivalence of personality adjective items across cultures. Journal of Research in Personality 2008, 42(6):1524-1536.
21. Rammstedt B., Goldberg L.R., Borg I. The measurement equivalence of Big-Five factor markers for persons with different levels of education. Journal of Research in Personality 2010, 44(1):53-61.
22. Riordan C.M., Vandenberg R.J. A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?. Journal of Management 1994, 20(3):643-671.
23. Schaffer B.S., Riordan C.M. A review of cross-cultural methodologies for organizational research: A best-practice approach. Organizational Research Methods 2003, 6(2):169-215.
24. Sprouse J. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods 2011, 43(1):155-167.
25. Stark S., Chernyshenko O.S., Drasgow F. Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology 2006, 91(6):1292-1306.
26. Steenkamp J.B.E.M., Baumgartner H. Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research 1998, 25(1):78-107.
27. Sun Y., Wang N., Peng Z. Working for one penny: Understanding why people would like to participate in online tasks with low payment. Computers in Human Behavior 2011, 27(2):1033-1041.
28. Vandenberg R.J., Lance C.E. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2000, 3(1):4-69.
29. Walsh B. Pennies for your thoughts. Time 2011, 177(4):55-56.