Can subject matter experts' ratings of statement extremity be used to streamline the development of unidimensional pairwise preference scales?

Organizational Research Methods

Volumn 14, Issue 2, 2011, Pages 256-278

  Stark, Stephen ^a   Chernyshenko, Oleksandr S ^b   Guenole, Nigel ^c  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

bootstrapping; computer simulation procedures; item response theory; measurement models; Monte Carlo; quantitative research

Indexed keywords

EID: 79953331053     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428109356712     Document Type: Article

References (47)

1. Arthur, W., Jr., Bell, S.T., Villado, A.J., & Doverspike, D. (2006). The use of person-organization fit in employment decision making: An assessment of its criterion-related validity. Journal of Applied Psychology, 91, 786-801.
2. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.
3. Bock, R.D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
4. Borman, W.C., Buck, D.E., Hanson, M.A., Motowidlo, S.J., Stark, S., & Drasgow, F. (2001). An examination of the comparative, reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
5. Borman, W.C. & Motowidlo, S.J. (1997). Task performance and contextual performance: The meaning for personnel selection research. Human Performance, 10, 99-109.
6. Campbell, J.P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed., pp. 687-731). Palo Alto, CA: Consulting Psychologists Press.
7. Campbell, J.P., Dunnette, M.D., Arvey, R.D., & Hellervik, L.V. (1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57, 15-22.
8. Chernyshenko, O.S., Stark, S., Chan, K.Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
9. Chernyshenko, O.S., Stark, S., Drasgow, F., & Roberts, B.W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88-106.
10. Chernyshenko, O.S., Stark, S., & Williams, A. (2009). Latent trait theory approach to measuring person-organization fit: Conceptual rationale and empirical evaluation. International Journal of Testing, 9, 358-380.
11. Chuah, S.C., Drasgow, F., & Luecht, R. (2006). How big is big enough? Sample size requirements for CAST item parameter estimation. Applied Measurement in Education, 19, 241-255.
12. Drasgow, F., Levine, M.V., Tsien, S., Williams, B.A., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.
13. Drasgow, F., & Olson-Buchanan, J.B. (Eds.). (1999). Innovations in computerized assessment. Mahwah, NJ: Lawrence Erlbaum.
14. Earley, P.C., & Ang, S. (2003). Cultural intelligence: Individual interactions across cultures. Stanford, CA: Stanford University Press.
15. Griffin, M.A., Neal, A., & Neale, M. (2000). The contribution of task performance and contextual performance to effectiveness: Investigating the role of situational constraints. Applied Psychology: An International Review, 49, 517-533.
16. Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff Publishing.
17. Hanisch, K.A., & Hulin, C.L. (1991). General attitudes and organizational withdrawal: An evaluation of a causal model. Journal of Vocational Behavior, 39, 110-128.
18. Harter, J.K., Schmidt, F.L., & Hayes, T.L. (2002). Business unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta analysis. Journal of Applied Psychology, 87, 268-279.
19. Hedge, J.W., Borman, W.C., Bruskiewicz, K.T., & Bourne, M.J. (2004). The development of an integrated performance category system for supervisory jobs in the U.S. Navy. Military Psychology, 16, 231-243.
20. Hogan, R. (2005). In defense of personality measurement: New wine for old whiners. Human Performance, 18, 331-341.
21. Houston, J.S., Borman, W.C., Farmer, W.L., & Bearden, R.M. (2005). Development of the enlisted computer adaptive personality scales (ENCAPS), renamed Navy Computer Adaptive Personality Scales (NCAPS). Technical Report No. 502. Navy Personnel Research, Studies, and Technology (NPRST). Navy Personnel Command.
22. Landy, F.J., Farr, J.L., Saal, F.E., & Freytag, W.R. (1976). Behaviorally anchored scales for rating the performance of police officers. Journal of Applied Psychology, 61, 750-758.
23. Law, K.S., Wong, C.S., & Song, L.J. (2004). The construct and criterion validity of emotional Intelligence and its potential utility for management studies. Journal of Applied Psychology, 89, 483-496.
24. Legree, P.J., Psotka, J., Tremble, T.R., & Bourne, D. (2005). Using consensus based measurement to assess emotional intelligence. In R. Schulze & R. Roberts (Eds.), International handbook of emotional intelligence (pp. 99-123). Berlin: Hogrefe & Huber.
25. Lievens, F., & Sackett, P.R. (2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 9, 1181-1188.
26. McDaniel, M.A., & Nguyen, N.T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
27. Mills, C. N., Potenza, M. T., Fremer, J. J., & Ward, W. C. (Eds.). (2002). Computer-based testing: Building the foundation for future assessments. Mahwah, NJ: Lawrence Erlbaum.
28. Organ, D.W. (1988). Organizational citizenship behavior: The good soldier syndrome. Lexington, MA: Lexington Books.
29. Roberts, B. (2001). Study behavior questionnaire. Unpublished manuscript. Department of Psychology, University of Illinois, Urbana-Champaign.
30. Roberts, B.W., Chernyshenko, O., Stark, S., & Goldberg, L. (2005). The structure of conscientiousness: An empirical investigation based on seven major personality questionnaires. Personnel Psychology, 58, 103-139.
31. Sackett, P.R., Berry, C.M., Wiemann, S.A., & Laczo, R.M. (2006). Citizenship and counterproductive behavior: Clarifying the relationships between the two domains. Human Performance, 19, 441-464.
32. Sackett, P.R., Schmitt, N., Ellingson, J.E., & Kabin, M.B. (2001). Highstakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative action world. American Psychologist, 56, 302-318.
33. Sireci, S. (1995). Using subject-matter experts to assess content representation: An MDS analysis. Applied Psychological Measurement, 19, 241-265.
34. Smith, P.C., & Kendall, L.M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
35. Stark, S. (2006a). MODFIT 2.0: An Excel VBA program for examining model-data fit with dichotomous and polytomous IRT models. Unpublished manuscript. University of South Florid.
36. Stark, S. (2006b). ZG-EAP: A computer program for EAP scoring with the Zinnes-Griggs model. Unpublished manuscript. University of South Florida.
37. Stark, S., Chernyshenko, O.S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise preference model. Applied Psychological Measurement, 29, 184-201.
38. Stark, S., & Drasgow, F. (1998). Application of an IRT ideal point model to computer adaptive assessment of job performance. Paper presented at the 13th annual conference for the Society of Industrial and Organizational Psychology. Dallas, TX.
39. Stark, S., & Drasgow, F. (2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26, 208-227.
40. Thissen, D., & Wainer, H. (Eds.). (2001). Test scoring. Mahwah, NJ: Lawrence Erlbaum.
41. Thurstone, L.L. (1927). Psychophysical analysis. American Journal of Psychology, 38, 368-389.
42. Vickers, R.R., Jr., Conway, T.L., & Hervig, L.K. (1990). Demonstration of replicable dimensions of health behaviors. Preventive Medicine, 19, 377-401.
43. Viswesvaran, C., Deller, J., & Ones, D.S. (2007). Personality measures in personnel selection: Some new contributions. International Journal of Selection and Assessment, 15, 354-358.
44. Wainer, H. (Ed.). (2000). Computerized-adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
45. Wanous, J.P., Reichers, A.E., & Hudy, M.J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82, 247-252.
46. Weekley, J.A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
47. Zinnes, J.L., & Griggs, R.A. (1974). Probabilistic, multidimensional unfolding analysis. Psychometrika, 39, 327-350.