Constructing Fake-Resistant Personality Tests Using Item Response Theory: High-Stakes Personality Testing with Multidimensional Pairwise Preferences

New Perspectives on Faking in Personality Assessment

Volumn , Issue , 2011, Pages

  Stark, Stephen ^a   Chernyshenko, Oleksandr S ^b   Drasgow, Fritz ^c  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

cAT; Computerized adaptive testing; Faking; Forced choice; High stakes; IRT; Item response theory; Pairwise preference; Personality

Indexed keywords

EID: 84921259092     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1093/acprof:oso/9780195387476.003.0061     Document Type: Chapter

References (68)

1. Andrich, D. (1995). Hyperbolic cosine latent trait models for unfolding direct responses and pairwise preferences. Applied Psychological Measurement, 19, 269-290.
2. Barrick, M., Mount, M. K, & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next?. International Journal of Selection & Assessment, 9, 9-30.
3. Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92, 410-424.
4. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.
5. Böckenholt, U. (2004). Comparative judgments as an alternative to ratings: Identifying the scale origin. Psychological Methods, 9, 453-465.
6. Bogg, T., & Roberts, B. W. (2004). Conscientiousness and health behaviors: A meta-analysis. Psychological Bulletin, 130, 887-919.
7. Booth-Kewley, S., & Vickers, R. R., Jr. (1994). Associations between major domains of personality and health behavior. Journal of Personality, 62, 281-298.
8. Borman, W. C., Buck, D., Hanson, M. A., Motowidlo, S. J., Stark, S., & Drasgow, F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
9. Bowen, C., Martin, B. A., & Hunt, S. T. (2002). A comparison of ipsative and normative approaches for the ability to control faking in personality questionnaires. International Journal of Organizational Analysis, 10, 240-259.
10. Butcher, J. N., Dahlstrom, W. G., Graham, I. R., Tellegen, A., & Kaemmer, B. (1989). MMPI-2: Minnesota Multiphasic Personality Inventory-2: Manual for administration and scoring. Minneapolis: University of Minnesota Press.
11. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
12. Chernyshenko, O. S., Stark, S., Chan, K. Y., Drasgow, F., & Williams, B. A. (2001). Fitting item response theory models to personality data. Multivariate Behavioral Research, 36, 523-562.
13. Chernyshenko, O. S., Stark, S., & Drasgow, F. (2011). Individual differences, their measurement, and validity. In S. Zedeck (Ed.), Handbook of industrial and organizational psychology. Washington, D.C.: American Psychological Association.
14. Chernyshenko, O. S., Stark, S., Drasgow, F., & Roberts, B. W. (2007a). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88-106.
15. Chernyshenko, O. S., Stark, S., Drasgow, F., Hulin, C. L., & Lopez-Rivas, G. E. (2007b). Behavioral domains assessed by TAPAS. Technical Report No. 2007-009. Urbana, IL: Drasgow Consulting Group.
16. Chernyshenko, O. S., Stark, S., Prewett, M. S., Gray, A. A., Stilson, F. R., & Tuttle, M. D. (2009). Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats. Human Performance, 22, 1-23.
17. Chernyshenko, O. S., Stark, S., Woo, S. E., & Conz, G. (2008). Openness to experience: Its facet structure, measurement, and usefulness in predicting important organizational outcomes. Paper presented at the 23rd annual conference for the Society of Industrial and Organizational Psychology, San Francisco, CA.
18. Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18, 267-307.
19. Dilchert, S., Ones, D. S., Viswesvaran, C., & Deller, J. (2006). Response distortion in personality measurement: Born to deceive, yet capable of providing valid self-assessments? Psychological Science, 48, 209-225.
20. Drasgow, F., Lee, W. C., Stark, S., & Chernyshenko, O.S. (Spring, 2001). Alternative methodologies for predicting attrition in the Army: The new AIM scales. (Report to HUMMRO.) Alexandria, VA.
21. Drasgow, F., & Levine, M. V. (1986). Optimal detection of certain forms of inappropriate test scores. Applied Psychological Measurement, 10, 59-67.
22. Drasgow, F., Levine, M. V., & Zickar, M. J. (1996). Optimal identification of mismeasured individuals. Applied Measurement in Education (Special Issue on Appropriateness Measurement), 9, 47-64.
23. Drasgow, F., Stark, S., & Chernyshenko, O.S. (2011). Tailored Adaptive Personality Assessment System (TAPAS) prediction of soldier performance. Paper presented at the 26th annual conference for the Society of Industrial and Organizational Psychology. Chicago, IL.
24. Dwight, S. A., & Donovan, J. J. (2003). Do warnings not to fake reduce faking? Human Performance, 16, 1-23.
25. Edwards, A. L. (1954). Personal preference schedule. New York: Psychological Corporation.
26. Ellingson, J. E., Sackett, P. R., & Hough, L. M. (1999). Social desirability corrections in personality measurement: Issues of applicant comparison and construct validity. Journal of Applied Psychology, 84, 155-166.
27. Foldes, H., Duehr, E. E., & Ones, D. S. (2008). Group difference in personality: Meta-analyses comparing five US racial groups. Personnel Psychology, 61, 579-616.
28. Guo, J., Tay, L., & Drasgow, F. (2009). Conspiracies and test compromise: An evaluation of the resistance of test systems to small-scale cheating. International Journal of Testing, 9, 283-309.
29. Heggestad, E. D., Morrison, M., Reeve, C. L., & McCloy, R. A. (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91, 9-24.
30. Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167-184.
31. Hogan, J., Rybicki, S. L., Motowidlo, S. J., & Borman, W. C. (1998). Relations between contextual performance, personality, and occupational advancement. Human Performance, 11, 189-207.
32. Hough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75, 95-108.
33. Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence, and lessons learned. International Journal of Selection and Assessment, 9, 152-194.
34. Jackson, D. N., Wrobleski, V. R., & Ashton, M. C. (2000). The impact of faking on employment tests: Does forcedchoice offer a solution? Human Performance, 13, 371-388.
35. Judge, T. A., Higgins, C. A., Thoresen, C. J., & Barrick, M. R. (1999). The big five personality traits, general mental ability, and career success across the life span. Personnel Psychology, 52, 621-652.
36. Kriedt, P. H., & Dawson, R. I. (1961). Response set and the prediction of clerical job performance. Journal of Applied Psychology, 45, 175-178.
37. Levine, M. V., & Drasgow, F. (1988). Optimal appropriateness measurement. Psychometrika, 53, 161-176.
38. McCloy, R. A., Heggestad, E. D., & Reeve, C. L. (2005). A silk purse from the sow's ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8, 222-248.
39. Meade, A. W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77, 531-552.
40. Meijer, R. R. (2003). Diagnosing item score patterns on a test using IRT based person-fit statistics. Psychological Methods, 8, 72-87.
41. Meijer, R. R. (2004). Using patterns of summed scores in paper-and-pencil tests and CAT to detect misfitting item score patterns. Journal of Educational Measurement, 41, 119-136.
42. Noftle, E. E., & Robins, R. W. (2007). Personality predictors of academic outcomes: Big five correlates of GPA and SAT scores. Journal of Personality and Social Psychology, 93, 116-130.
43. Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.
44. Roberts, J. S., Donoghue, J. R., & Laughlin, J. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3-32.
45. Robson, S. M., Jones, A., & Abraham, J. (2008). Personality, faking, and convergent validity: A warning concerning warning statements. Human Performance, 21, 89-106.
46. Rosse, J. G., Stecher, M., Miller, J. L., & Levin, R. A. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83, 634-644.
47. Rounds, J. B., Henly, G. A., Dawis, R. V., Lofquist, L. H., & Weiss, D. J. (1981). Manual for the Minnesota Importance Questionnaire. Minneapolis: University of Minnesota, Vocational Psychology Research.
48. Schmitt, N., & Oswald, F. L. (2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. Journal of Applied Psychology, 91, 613-621.
49. Stark, S. (2002a). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment. Doctoral dissertation. University of Illinois at Urbana-Champaign.
50. Stark, S. (2002b). Computer program for scoring multi-unidimensional pairwise preference (MUPP) tests. Unpublished manuscript. University of Illinois at Urbana-Champaign.
51. Stark, S. (2006). ZG-EAP: A computer program for EAP scoring with the Zinnes-Griggs model. Unpublished manuscript. University of South Florida.
52. Stark, S., & Chernyshenko, O. S. (June, 2007). Adaptive testing with the multi-unidimensional pairwise preference (MUPP) model. Proceedings of the 2007 Graduate Management Admissions Council conference on computerized adaptive testing, Minneapolis, MN.
53. Stark, S., Chernyshenko, O. S., Chan, K. Y., Lee, W. C., & Drasgow, F. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943-953.
54. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005a). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: An application to the problem of faking in personality assessment. Applied Psychological Measurement, 29, 184-201.
55. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005b). Examining the recovery of normative scores with three-and five-dimensional nonadaptive multidimensional forced choice tests. Paper presented at the 47th annual conference of the International Military Testing Association. Singapore.
56. Stark, S., Chernyshenko, O. S., Drasgow, F., White, L. A., Heffner, T., & Hunter, A. (2008). Using multidimensional pairwise preference personality tests in military contexts: Development and evaluation of the TAPAS-95s. Paper presented at the 50th annual conference of the International Military Testing Association. Amsterdam, The Netherlands.
57. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2010). Update on Tailored Adaptive Personality Assessment System (TAPAS): Results and ideas to meet the challenges of high stakes testing. Paper presented at the 52nd annual conference of the International Military Testing Association. Lucerne, Switzerland.
58. Stark, S., Chernyshenko, O. S., Drasgow, F., & Williams, B. A. (2006). Item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91, 25-39.
59. Stark, S., & Drasgow, F. (2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26, 208-227.
60. Stark, S., Drasgow, F., & Chernyshenko, O. S. (October 2008). Update on the Tailored Adaptive Personality Assessment System (TAPAS): The next generation of personality assessment systems to support personnel selection and classification decisions. Paper presented at the 50th annual conference of the International Military Testing Association, Amsterdam, The Netherlands.
61. Timmerman, T. A. (2006). Predicting turnover with broad and narrow personality traits. International Journal of Selection and Assessment, 14, 392-399.
62. Tross, S. A., Harper, J. P., Osher, L. W., & Kneidinger, L. M. (2000). Not just the usual cast of characteristics: Using personality to predict college performance and retention. Journal of College Student Development, 41, 323-334.
63. Tucker, J. S., Kressin, N. R., Spiro, A., & Ruscio, J. (1998). Intrapersonal characteristics and the timing of divorce: A prospective investigation. Journal of Social and Personal Relationships, 15, 211-225.
64. Vasilopoulos, N. L., Cucina, J. M., Dyomina, N. V., Morewitz, C. L., & Reilly, R. R. (2006). Forced-choice personality tests: A measure of personality and cognitive ability? Human Performance, 19, 175-199.
65. Villanova, P., Bernardin, H. J., Johnson, D. L., & Dahmus, S. A. (1994). The validity of a measure of job compatibility in the prediction of job performance and turnover of motion picture theater personnel. Personnel Psychology, 47, 73-90.
66. White, L. A., & Young, M. C. (1998). Development and validation of the Assessment of Individual Motivation (AIM). Paper presented at the Annual Meeting of the American Psychological Association, San Francisco, CA.
67. Zickar, M. J., & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71-87.
68. Zinnes, J. L., & Griggs, R. A. (1974). Probabilistic, multidimensional unfolding analysis. Psychometrika, 39, 327-350.