Adaptive Testing With Multidimensional Pairwise Preference Items: Improving the Efficiency of Personality and Other Noncognitive Assessments

Organizational Research Methods

Volumn 15, Issue 3, 2012, Pages 463-487

  Stark, Stephen ^a   Chernyshenko, Oleksandr S ^b   Drasgow, Fritz ^c   White, Leonard A ^d  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^d US Army Research Institute for the Behavioral and Social Sciences   (United States)

Author keywords

CAT; computerized adaptive testing; ideal point; IRT; item response theory; multidimensional forced choice; pairwise preference

Indexed keywords

EID: 84862238769     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428112444611     Document Type: Article

References (61)

1. Andrich D.Hyperbolic cosine latent trait models for unfolding direct responses and pairwise preferences.Applied Psychological Measurement. 1995;19:269-290.
2. Ashton M. C.,Lee K.,Vernon P. A.,Jang K. L.Fluid intelligence, crystallized intelligence, and the Openness/Intellect factor.Journal of Research in Personality. 2000;34:197-207.
3. Barrick M. R.,Mount M. K.The Big Five personality dimensions and job performance: A meta-analysis.Personnel Psychology. 1991;44:1-26.
4. Bartram D.Increasing validity with forced-choice criterion measurement format.International Journal of Selection and Assessment. 2007;15:263-272.
5. Böckenholt U.Comparative judgments as an alternative to ratings: Identifying the scale origin.Psychological Methods. 2004;9:453-465.
6. Borman W. C.,Buck D. E.,Hanson M. A.,Motowidlo S. J.,Stark S.,Drasgow F.An examination of the comparative, reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales.Journal of Applied Psychology. 2001;86:965-973.
7. Borman W. C.,Motowidlo S. J.Task performance and contextual performance: The meaning for personnel selection research.Human Performance. 1997;10:99-109.
8. Brown A.,Maydeu-Olivares A.Item response modeling of forced-choice questionnaires.Educational and Psychological Measurement. 2011;71:460-502.
9. Butcher J. N.,Dahlstrom W. G.,Graham I. R.,Tellegen A.,Kaemmer B.MMPI-2: Minnesota Multiphasic Personality Inventory-2: Manual for administration and scoring. Minneapolis: University of Minnesota Press; 1989:.
10. Campbell J. P.Handbook of industrial and organizational psychology. 1990). Modeling the performance prediction problem in industrial and organizational psychology. In DunnetteM.HoughL. M. (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed., pp. 687-731). Palo Alto, CA: Consulting Psychologists Press.Dunnette M.Hough L. M., ed. Palo Alto, CA: Consulting Psychologists Press; 1990:687-731.
11. Chatman J. A.Matching people and organizations: Selection and socialization in public accounting firms.Administrative Science Quarterly. 1991;36:459-484.
12. Chernyshenko O. S.Applications of ideal point approaches to scale construction and scoring in personality measurement: The development of a six-faceted measure of conscientiousness.Dissertation Abstracts International. 2002;63:5556.
13. Chernyshenko O. S.,Stark S.,Drasgow F.Handbook of industrial and organizational psychology. Zedeck S., ed. Washington, DC: American Psychological Association; 2010:117-151.
14. Chernyshenko O. S.,Stark S.,Drasgow F.,Roberts B. W.Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures.Psychological Assessment. 2007;19:88-106.
15. Chernyshenko O. S.,Stark S.,Prewett M. S.,Gray A. A.,Stilson F. R.,Tuttle M. D.Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats.Human Performance. 2009;22:105-127.
16. Cheung M. W. L.,Chan W.Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis.Structural Equation Modeling. 2002;9:55-77.
17. Christiansen N. D.,Burns G. N.,Montgomery G. E.Reconsidering forced-choice item formats for applicant personality assessment.Human Performance. 2005;18:267-307.
18. Dilchert S.,Ones D. S.,Viswesvaran C.,Deller J.Response distortion in personality measurement: Born to deceive, yet capable of providing valid self-assessments?.Psychological Science. 2006;48:209-225.
19. Drasgow F.,Levine M.V.Optimal detection of certain forms of inappropriate test scores.Applied Psychological Measurement. 1986;10:59-67.
20. Dwight S. A.,Donovan J. J.Do warnings not to fake reduce faking?.Human Performance. 2003;16:1-23.
21. Elosua P.Assessing vocational interests in the Basque Country using paired comparison design.Journal of Vocational Behavior. 2007;71:135-145.
22. Embretson S. E.,Reise S. P.Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum; 2000:.
23. Gordon L. V.Validities of the forced-choice and questionnaire methods of personality measurement.Journal of Applied Psychology. 1951;35:407-412.
24. Guo J.,Tay L.,Drasgow F.Conspiracies and test compromise: An evaluation of the resistance of test systems to small-scale cheating.International Journal of Testing. 2009;9:283-309.
25. Hanisch K. A.,Hulin C. L.General attitudes and organizational withdrawal: An evaluation of a causal model.Journal of Vocational Behavior. 1991;39:110-128.
26. Heggestad E. D.,Morrison M.,Reeve C. L.,McCloy R. A.Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance.Journal of Applied Psychology. 2006;91:9-24.
27. Higgins D. M.,Peterson J. B.,Pihl R. O.,Lee A. G. M.Prefrontal cognitive ability, intelligence, Big Five personality, and the prediction of advanced academic and workplace performance.Journal of Personality and Social Psychology. 2007;93:298-319.
28. Houston J. S.,Borman W. C.,Farmer W. L.,Bearden R. M.Development of the enlisted computer adaptive personality scales (ENCAPS), renamed Navy Computer Adaptive Personality Scales (NCAPS). Millington, TN: Navy Personnel Research, Studies, and Technology (NPRST). Navy Personnel Command; 2005:.
29. Hulin C. L.,Stark S.,Drasgow F.Psychometric solutions for the faking problem: Multidimensional pairwise preference IRT model. Arlington, VA: Army Research Institute; 2006:.
30. Jackson D. N.,Wrobleski V. R.,Ashton M. C.The impact of faking on employment tests: Does forced-choice offer a solution?.Human Performance. 2000;13:371-388.
31. Judge T. A.,Erez A.,Bono J. E.,Thoresen C.Are measures of self-esteem, neuroticism, locus of control, and generalized self-efficacy indicators of a common core construct?.Journal of Personality and Social Psychology. 2002;83:693-710.
32. Kriedt P. H.,Dawson R. I.Response set and the prediction of clerical performance.Journal of Applied Psychology. 1961;45:175-178.
33. Likert R.A technique for the measurement of attitudes.Archives of Psychology. 1932;22:1-55.
34. Maydeu-Olivares A.,Brown A.Item response modeling of paired comparison and ranking data.Multivariate Behavioral Research. 2010;45:935-974.
35. McCloy R. A.,Heggestad E. D.,Reeve C. L.A silk purse from the sow's ear: Retrieving normative information from multidimensional forced-choice items.Organizational Research Methods. 2005;8:222-248.
36. Meade A. W.Psychometric problems and issues involved with creating and using ipsative measures for selection.Journal of Occupational and Organizational Psychology. 2004;77:531-552.
37. Ones D. S.,Viswesvaran C.,Schmidt F. L.Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance.Journal of Applied Psychology. 1993;78:679-703.
38. Organ D. W.,Ryan K.A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behavior.Personnel Psychology. 1995;48:775-802.
39. Paunonen S. V.,Jackson D. N.What is beyond the Big Five? Plenty!.Journal of Personality. 2000;68:821-835.
40. Press W. H.,Flannery B. P.,Teukolsky S. A.,Vetterling W. T.Numerical recipes: The art of scientific computing. New York, NY: Cambridge University Press; 1990:.
41. Pulakos E. D.,Arad S.,Donovan M. A.,Plamondon K. E.Adaptability in the workplace: Development of a taxonomy of adaptive performance.Journal of Applied Psychology. 2000;85:612-624.
42. Roberts B.,Chernyshenko O. S.,Stark S.,Goldberg L.The construct of conscientiousness: The convergence between lexical models and scales drawn from six major personality questionnaires.Personnel Psychology. 2005;58:103-139.
43. Roberts J. S.,Donoghue J. R.,Laughlin J. E.A general item response theory model for unfolding unidimensional polytomous responses.Applied Psychological Measurement. 2000;24:3-32.
44. Sackett P. R.,Berry C. M.,Wiemann S. A.,Laczo R. M.Citizenship and counterproductive behavior: Clarifying the relationships between the two domains.Human Performance. 2006;19:441-464.
45. Sackett P. R.,Schmitt N.,Ellingson J. E.,Kabin M. B.High stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative action world.American Psychologist. 2001;56:302-318.
46. Schmitt N.,Oswald F. L.The impact of corrections for faking on the validity of noncognitive measures in selection settings.Journal of Applied Psychology. 2006;91:613-621.
47. Schneider B.,Goldstein H. W.,Smith D. B.The ASA framework: An update.Personnel Psychology. 1995;48:747-779.
48. OPQ32: Technical manual. Thames Ditton, UK: SHL Group plc; 2006:.
49. Stark S.A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment: The generalized graded unfolding model for multi-unidimensional paired comparison responses.Dissertation Abstracts International. 2002;63:1084.
50. Stark S.,Chernyshenko O. S.Computerized adaptive testing with the Zinnes and Griggs pairwise preference ideal point model.International Journal of Testing. 2011;11:231-247.
51. Stark S.,Chernyshenko O. S.,Chan K. Y.,Lee W. C.,Drasgow F.Effects of the testing situation on item responding: Cause for concern.Journal of Applied Psychology. 2001;86:943-953.
52. Stark S.,Chernyshenko O. S.,Drasgow F.An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: An application to the problem of faking in personality assessment.Applied Psychological Measurement. 2005;29:184-201.
53. Stark S.,Chernyshenko O. S.,Drasgow F.New perspectives on faking in personality assessment. Ziegler M.MacCann C.Roberts R., ed. London: Oxford University Press; 2011:214-239.
54. Stark S.,Chernyshenko O. S.,Drasgow F.,Williams B. A.Item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?.Journal of Applied Psychology. 2006;91:25-39.
55. Stark S.,Drasgow F.An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model.Applied Psychological Measurement. 2002;26:208-227.
56. Stark S.,Drasgow F.,Chernyshenko O. S.Update on the Tailored Adaptive Personality Assessment System (TAPAS): The next generation of personality assessment systems to support personnel selection and classification decisions. 2008;:.
57. Tett R. P.,Jackson D. N.,Rothstein M.Personality measures as predictors of job performance: A meta-analytic review.Personnel Psychology. 1991;44:703-742.
58. van der Linden W. J.,Glas C. A.Computerized adaptive testing: Theory and practice. Boston, MA: Kluwer; 2000:.
59. Vasilopoulos N. L.,Cucina J. M.,Dyomina N. V.,Morewitz C. L.,Reilly R. R.Forced-choice personality tests: A measure of personality and cognitive ability?.Human Performance. 2006;19:175-199.
60. White L. A.,Young M. C.Development and validation of the Assessment of Individual Motivation (AIM). San Francisco, CA: Paper presented at the Annual Meeting of the American Psychological Association; 1998:.
61. Zickar M. J.,Drasgow F.Detecting faking on a personality instrument using appropriateness measurement.Applied Psychological Measurement. 1996;20:71-87.