Outliers in questionnaire data: Can they be detected and should they be removed?

Journal of Educational and Behavioral Statistics

Volumn 36, Issue 2, 2011, Pages 186-212

  Zijlstra, Wobbe P ^a   van der Ark, L Andries ^a   Sijtsma, Klaas ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

contaminant observations; discordancy testing; outlier detection in questionnaire data; outlier detection methods for questionnaire data; suspected patterns of item scores

Indexed keywords

EID: 79951776134     PISSN: 10769986     EISSN: None     Source Type: Journal    
DOI: 10.3102/1076998610366263     Document Type: Article

References (41)

1. Atkinson, A.C., & Riani, M. (2000). Robust diagnostic regression analysis. New York, NY: Springer.
2. Austin, E.J., Deary, I.J., Gibson, G.J., McGregor, M.J., & Dent, J.B. (1998). Individual response spread in self-reported scales: Personality correlations and consequences. Personality and Individual Differences, 24, 421-438.
3. Bachman, J.G., & O'Malley, P.M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491-509.
4. Barnett, V., & Lewis, T. (1994). Outliers in statistical data (3rd ed.). New York, NY: John Wiley.
5. Barnette, J.J. (1999). Nonattending respondent effects on internal consistency of self-administered surveys: A Monte Carlo simulation study. Educational and Psychological Measurement, 59, 38-46.
6. Baumeister, R.F., & Tice, D.M. (1988). Metatraits. Journal of Personality, 56, 571-598.
7. Breunig, M.M., Kriegel, H.-P., Ng, R.T., & Sander, J. (2000). LOF: Identifying density-based local outliers. SIGMOD Record, 29, 93-104.
8. Chatterjee, S., & Hadi, A.S. (1986). Influential observations, high leverage points, and outliers in linear regression. Statistical Science, 1, 379-416.
9. Christmann, A., & Van Aelst, S. (2006). Robust estimation of Cronbach's alpha. Journal of Multivariate Analysis, 97, 1660-1674.
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11. Costa, P.T., & McCrae, R.R. (1992). The NEO personality inventory and the NEO five factor inventory professional manual. Odessa, FL: Psychological Assessment Resources.
12. Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
13. Denollet, J. (1994). Health complaints and outcome assessment in coronary heart disease. Psychosomatic Medicine, 56, 463-474.
14. Denollet, J. (2005). DS14: Standard assessment of negative affectivity, social inhibition, and type D personality. Psychosomatic Medicine, 67, 89-97.
15. Devlin, S.J., Gnanadesikan, R., & Kettenring, J.R. (1975). Robust estimation and outlier detection with correlational coefficients. Biometrika, 62, 531-545.
16. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
17. Emons, W.H.M. (2008). Person-fit analysis of polytomous items. Applied Psychological Measurement, 32, 224-247.
18. Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction (pp. 60-90). Princeton, NJ: Princeton University Press.
19. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., & Stahel, W.A. (1986). Robust statistics: The approach based on influence functions. New York, NY: John Wiley.
20. Hawkins, D.M. (1980). Identification of outliers. London, England: Chapman and Hall.
21. Kaufman, L., & Rousseeuw, P.J. (2005). Finding groups in data: An introduction to cluster analysis. Hoboken, NJ: John Wiley.
22. Mahalanobis, P.C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2, 49-55.
23. Maxwell, S.E., & Delaney, H.D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
24. McFarland, L.A., & Ryan, A.M. (2000). Variance in faking across noncognitive measures. Journal of Applied Psychology, 85, 812-821.
25. Meijer, R.R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18, 311-314.
26. Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
27. Mischel, W. (1968). Personality and assessment. New York, NY: John Wiley.
28. Molenaar, I.W. (1991). A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items. Kwantitatieve Methoden, 12(37), 97-117.
29. Molenaar, I.W. (1997). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of item response theory (pp. 369-380). New York, NY: Springer.
30. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
31. Rosner, B. (1983). Percentage points for a generalized ESD many-outlier procedure. Technometrics, 25, 165-172.
32. Rousseeuw, P.J., & Leroy, A.M. (2003). Robust regression and outlier detection. Hoboken, NJ: John Wiley.
33. Samejima, F. (1997). Graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of item response theory (pp. 85-100). New York, NY: Springer.
34. Selvin, S. (2004). Statistical analysis of epidemiological data. New York, NY: Oxford University Press.
35. Tukey, J.W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
36. Van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of item response theory. New York, NY: Springer.
37. Van Ginkel, J.R., & Van der Ark, L.A. (2005). SPSS syntax for missing value imputation in test and questionnaire data. Applied Psychological Measurement, 29, 152-153.
38. Yick, J.S., & Lee, A.H. (1998). Unmasking outliers in two-way contingency tables. Computational Statistics & Data Analysis, 29, 69-79.
39. Zickar, M.J., & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71-87.
40. Zijlstra, W.P., Van der Ark, L.A., & Sijtsma, K. (2007). Outlier detection in test and questionnaire data. Multivariate Behavioral Research, 42, 531-555.
41. Zijlstra, W.P., Van der Ark, L.A., & Sijtsma, K. (2010). Discordancy tests for outlier detection in multi-item surveys. Manuscript submitted for publication.