Methods for the detection of carelessly invalid responses in survey data

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 4-19

  Curran, Paul G ^a  

^a KENYON COLLEGE   (United States)

Author keywords

Careless responding; Data cleaning; Insufficient effort responding; Invalid response

Indexed keywords

EID: 84939824501     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.07.006     Document Type: Article

References (46)

1. Beach, D.A., Identifying the random responder. The Journal of Psychology 123 (1989), 101–103.
2. Berry, D.T.R., Baer, R.A., Harris, M.J., Detection of malingering on the MMPI: A meta-analysis. Clinical Psychology Review 11 (1991), 585–598.
3. Birkeland, S.A., Manson, T.M., Kisamore, J.L., Brannick, M.T., Smith, M.A., A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment 14 (2006), 317–335.
4. Burns, G.N., Christiansen, N.D., Morris, M.B., Periard, D.A., Coaster, J.A., Effects of applicant personality on resume evaluations. Journal of Business and Psychology, 2014, 1–19.
5. Butcher, J.N., Dahlstrom, W.G., Graham, J.R., Tellegen, A., Kaemmer, B., Minnesota multiphasic personality inventory 2 (MMPI-2): Manual for administration and scoring. 1989, University of Minnesota Press, Minneapolis, MN.
6. Costa, P.T. Jr., McCrae, R.R., The revised NEO personality inventory (NEO-PI-R). Saklofske, D.H., (eds.) The SAGE handbook of personality theory and assessment Personality measurement and testing, 2, 2008, Sage, Thousand Oaks, CA, 179–198.
7. Curran, P.G., Hauser, K.A., Understanding responses to check items: A verbal protocol analysis. Paper presented at the 30th Annual Conference of the Society for Industrial and Organizational Psychology, Philadelphia, PA, April 23–25, 2015, 2015.
8. Curran, P., Kotrba, L.M., The impacts of invalid responding: a simulation study. Presented at the 27th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA, April 26–28, 2012, 2012.
9. Curran, P., Kotrba, L., Denison, D., Careless responding in surveys: Applying traditional techniques to organizational settings. Poster presented at the 25th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA, April 8–10, 2010, 2010.
10. Davey, T., Nering, M., Controlling item exposure and maintaining item security. Mills, C.N., Potenza, M.T., Fremer, J.J., Ward, W.C., (eds.) Computer-based testing: Building the foundation for future assessments, 2002, 165–191.
11. DeRight, J., Jorgensen, R.S., I just want my research credit: Frequency of suboptimal effort in a non-clinical healthy undergraduate sample. The Clinical Neuropsychologist 29 (2015), 101–117.
12. Desimone, J.A., Harms, P.D., Desimone, A.J., Best practice recommendations for data screening. Journal of Organizational Behavior 36 (2015), 171–181.
13. Donlon, T.F., Fischer, F.E., An index of an individual's agreement with group-determined item difficulties. Educational and Psychological Measurement 28 (1968), 105–113.
14. Donnellan, M.B., Oswald, F.L., Baird, B.M., Lucas, R.E., The mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment 18 (2006), 192–203.
15. Dunnette, M.D., McCartney, J., Carlson, H.C., Kirchner, W.K., A study of faking behavior on a forced choice self-description checklist. Personnel Psychology 15 (1962), 13–24.
16. Ehlers, C., Greene-Shortridge, T.M., Weekley, J.A., Zajack, M.D., The exploration of statistical methods in detecting random responding. Paper presented at the annual meeting of the Society for Industrial/Organizational Psychology, Atlanta, GA, 2009.
17. Emons, W.H.M., Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement 32 (2008), 224–247.
18. Goldberg, L.R., Kilkowski, J.M., The prediction of semantic consistency in self descriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology 48 (1985), 82–98.
19. Gough, H.G., Bradley, P., CPI manual: Third edition. Palo Alto, CA: Consulting Psychologists Press, 1996.
20. Griffith, R.L., Peterson, M.H., (eds.) A closer examination of applicant faking behavior, 2006, Information Age Publishing, Greenwich, CT.
21. Guttman, L., A basis for scaling qualitative data. American Sociological Review 9 (1944), 139–150.
22. Guttman, L., The basis for scalogram analysis. Stouffer, S.A., Guttman, L., Suchman, E.A., Lazarsfeld, P.F., Star, S.A., Claussen, J.A., (eds.) Measurement and Prediction, 1950, Princeton University Press, Princeton, 66–90.
23. Hauser, D.J., Schwartz, N., Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 2015 (in press).
24. Head, M.L., Holman, L., Lanfear, R., Kahn, A.T., Jennions, M.D., The extent and consequences of p-hacking in science. PLoS Biology, 13, 2015.
25. Huang, J.L., Bowling, N.A., Liu, M., Li, Y., Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology 30 (2014), 299–311.
26. Huang, J.L., Curran, P.G., Keeney, J., Poposki, E.M., DeShon, R.P., Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology 27 (2012), 99–114.
27. Huang, J.L., Liu, M., Bowling, N.A., Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology 100 (2015), 828–845.
28. Jackson, D.N., The appraisal of personal reliability. Paper presented at the meetings of the Society of Multivariate Experimental Psychology, 1976, University Park, PA.
29. Jackson, D.N., Jackson vocational interest survey manual. 1977, Research Psychologists Press, Port Huron, MI.
30. Johnson, J.A., Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality 39 (2005), 103–129.
31. Karabatsos, G., Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education 16 (2003), 277–298.
32. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B. Jr., Bahník, Š., Bernstein, M.J., et al. Investigating variation in replicability. Social Psychology 45 (2014), 142–152.
33. Klein, R.A., Vianelli, M., Ratliff, K.A., Attention checking the “Many Labs” participants: Did participant attention moderate the included effects?. Poster presented at the Sixteenth Annual Meeting of the Society for Personality and Social Psychology, Long Beach, CA, 2015.
34. Maesschalck, D., Jouan-Rimbaud, D.L., Massart, D.L., The Mahalanobis distance. Chemometrics and Intelligent Laboratory Systems 50 (2000), 1–18.
35. Mahalanobis, P.C., On the generalized distance in statistics. Proceedings of the National Institute of Science of India 12 (1936), 49–55.
36. Maniaci, M.R., Rogge, R.D., Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality 48 (2014), 61–83.
37. Marjanovic, Z., Holden, R., Struthers, W., Cribbie, R., Greenglass, E., The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders. Personality and Individual Differences 84 (2015), 79–83.
38. Meade, A.W., Craig, S.B., Identifying careless responses in survey data. Psychological Methods 17 (2012), 437–455.
39. Oppenheimer, D.M., Meyvis, T., Davidenko, N., Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 45 (2009), 867–872.
40. Orpen, C., The fakability of the Edwards Personal Preference Schedule in personnel selection. Personnel Psychology 24 (1971), 1–4.
41. Peck, R., Devore, J.L., Statistics: The exploration & analysis of data. 2012, Brooks/Cole, Boston, MA.
42. Schmitt, N., Stults, D.M., Factors defined by negatively keyed items: The result of careless respondents?. Applied Psychological Measurement 9 (1985), 367–373.
43. Sijtsma, K., Molenaar, I.W., Introduction to nonparametric item response theory. 2002, Sage, Thousand Oaks, CA.
44. Sonderen, E.V., Sanderman, R., Coyne, J.C., Ineffectiveness of reverse wording of questionnaire items: Let's learn from cows in the rain. PLoS ONE, 8, 2013, e68967, 10.1371/journal.pone.0068967.
45. Spector, P.E., Van Katwyk, P.T., Brannick, M.T., Chen, P.Y., When two factors don't reflect two constructs: How item characteristics can produce artificial factors. Journal of Management 23 (1997), 659–677.
46. Woods, C.M., Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment 28 (2006), 189–194.