Preventing Rater Biases in 360-Degree Feedback by Forcing Choice

Organizational Research Methods

Volumn 20, Issue 1, 2017, Pages 121-148

  Brown, Anna ^a   Inceoglu, Ilke ^b   Lin, Yin ^a,c  

^a UNIVERSITY OF KENT   (United Kingdom)

^b UNIVERSITY OF SURREY   (United Kingdom)

^c CEB SHL Talent Measurement Solutions ^*  (United Kingdom)

Author keywords

forced choice; halo effect; ideal employee factor; multisource feedback; rater biases; Thurstonian IRT model

Indexed keywords

EID: 85002488059     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428116668036     Document Type: Article

References (66)

1. Adler S., Campion M., Colquitt A., Grubb A., Murphy K., Ollander-Krane R., Pulakos E. D., (2016). Getting rid of performance ratings: Genius or folly? A debate. Industrial and Organizational Psychology, 9(2), 219–252. doi:10.1017/iop.2015.106
2. Asparouhov T., (2006). General multi-level modeling with sampling weights. Communications in Statistics: Theory and Methods, 35, 439–460. doi:10.1080/03610920500476598
3. Asparouhov T., Muthén B., (2010). Multiple imputation with Mplus (Version 2). Retrieved from http://www.statmodel.com
4. Barr M. A., Raju N. S., (2003). IRT-based assessments of rater effects in multiple-source feedback instruments. Organizational Research Methods, 6(1), 15–43. doi:10.1177/1094428102239424
5. Bartram D., (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272. doi:10.1111/j.1468-2389.2007.00386.x
6. Bartram D., Brown A., Fleck S., Inceoglu I., Ward K., (2006). OPQ32 technical manual. Surrey, UK: SHL Group.
7. Bartram D., Robertson I., Callinan M., (2002). A framework for examining organizational effectiveness. In, Robertson I. T., Callinan M., Bartram D., (Eds.), Organizational effectiveness: The role of psychology (pp. 227–255). New York, NY: John Wiley & Sons.
8. Böckenholt U., (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665–678. doi:10.1037/a0028111
9. Böckenholt U., (2014). Modeling motivated misreports to sensitive survey questions. Psychometrika, 79(3), 515–537. doi:10.1007/s11336-013-9390-9
10. Bolt D. M., Lu Y., Kim J. S., (2014). Measurement and control of response styles using anchoring vignettes: A model-based approach. Psychological Methods, 19(4), 528–541. doi:10.1037/met0000016
11. Bolt D. M., Newton J. R., (2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814–833. doi:10.1177/0013164410388411
12. Borman W. C., (1997). 360 degree ratings: An analysis of assumptions and a research agenda for evaluating their validity. Human Resource Management Review, 7, 299–316. doi:10.1016/S1053-4822(97)90010-3
13. Bozeman D. P., (1997). Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior, 18, 313–316.
14. Brown A., (2016). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135–160. doi:10.1007/s11336-014-9434-9
15. Brown A., Bartram D., (2009-2011). OPQ32r technical manual. Surrey, UK: SHL Group.
16. Brown A., Maydeu-Olivares A., (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460–502. doi:10.1177/0013164410375112
17. Brown A., Maydeu-Olivares A., (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44, 1135–1147. doi:10.3758/s13428-012-0217-x
18. Brown A., Maydeu-Olivares A., (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. doi:10.1037/a0030641
19. Cheung M. W. L., Chan W., (2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling, 9, 55–77. doi:10.1207/S15328007SEM0901_4
20. Conway J. M., Huffcutt A. I., (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331–360. doi:10.1207/s15327043hup1004_2
21. Eckert R., Ekelund B. Z., Gentry W. A., Dawson J. F., (2010). “I don’t see me like you see me, but is that a problem?” Cultural influences on rating discrepancy in 360-degree feedback instruments. European Journal of Work and Organizational Psychology, 19(3), 259–278. doi:10.1080/13594320802678414
22. Eden D., Leviatan U., (1975). Implicit leadership theory as a determinant of the factor structure underlying supervisory behavior scales. Journal of Applied Psychology, 60(6), 736–741. doi:10.1037/0021-9010.60.6.736
23. Hansbrough T. K., Lord R., Schyns B., (2015). Reconsidering the accuracy of follower leadership ratings. The Leadership Quarterly, 26(2), 220–237. doi:10.1016/j.leaqua.2014.11.006
24. Heggestad E. D., Morrison M., Reeve C. L., McCloy R. A., (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91, 9–24. doi:10.1037/0021-9010.91.1.9
25. Hogan R., Shelton D., (1998). A socioanalytic perspective on job performance. Human Performance, 11(2), 129–144. doi:10.1207/s15327043hup1102&3_2
26. Hox J. J., (2010). Multilevel analysis: Techniques and applications (2nd ed.). London: Routledge.
27. Hu L., Bentler P. M., (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118
28. Judge T. A., Piccolo R. F., (2004). Transformational and transactional leadership: A meta-analytic test of their relative validity. Journal of Applied Psychology, 89, 755–768. doi:10.1037/0021-9010.89.5.755
29. Kahneman D., (2011). Thinking, fast and slow. London: Allen Lane.
30. Klehe U. C., Kleinmann M., Hartstein T., Melchers K. G., König C. J., Heslin P. A., Lievens F., (2012). Responding to personality tests in a selection context: The role of the ability to identify criteria and the ideal-employee factor. Human Performance, 25(4), 273–302. doi:10.1080/08959285.2012.703733
31. Landy F. J., Farr J. L., (1980). Performance rating. Psychological Bulletin, 87, 72–107. doi:10.1037/0033-2909.87.1.72
32. Landy F. J., Vance R. J., Barnes-Farrell J. L., Steele J. W, (1980). Statistical control of halo error in performance ratings. Journal of Applied Psychology, 65, 501–506. doi:10.1037/0021-9010.65.5.501
33. LeBreton J. M., Burgess J. R., Kaiser R. B., Atchley E. K., James L. R., (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6(1), 80–128. doi:10.1177/1094428102239427
34. Lorenzo-Seva U., Ferrando P. J., (2013). FACTOR 9.2: A comprehensive program for fitting exploratory and semiconfirmatory factor analysis and IRT models. Applied Psychological Measurement, 37(6), 497–498. doi:10.1177/0146621613487794
35. MacCallum R. C., Browne M. W., Sugawara H. M., (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. doi:10.1037/1082-989X.1.2.130
36. Maydeu-Olivares A., (1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika, 64, 325–340. doi:10.1007/BF02294299
37. Maydeu-Olivares A., Brown A., (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45, 935–974. doi:10.1177/0013164410375112
38. Maydeu-Olivares A., Coffman D. L., (2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362. doi:10.1037/1082-989x.11.4.344
39. Meade A. W., Craig S. B., (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. doi:10.1037/a0028085
40. Murphy K. R., Balzer W. K., (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619–624. doi:10.1037/0021-9010.74.4.619
41. Murphy K. R., Cleveland J. N., (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
42. Murphy K. R., Cleveland J. N., Skattebo A. L., Kinney T. B., (2004). Raters who pursue different goals give different ratings. Journal of Applied Psychology, 89(1), 158–164. doi:10.1037/0021-9010.89.1.158
43. Murphy K. R., Jako R. A., Anhalt R. L., (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218–225. doi:10.1037/0021-9010.78.2.218
44. Muthén L. K., Muthén B. O., (1998-2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
45. Podsakoff P. M., MacKenzie S. B., Lee J. Y., Podsakoff N. P., (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. doi:10.1037/0021-9010.88.5.879
46. Randall R., Sharples D., (2012). The impact of rater agreeableness and rating context on the evaluation of poor performance. Journal of Occupational and Organizational Psychology, 85(1), 42–59. doi:10.1348/2044-8325.002002
47. Rindskopf D., Rose T., (1988). Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research, 23, 51–67. doi:10.1207/s15327906mbr2301_3
48. Rush M. C., Thomas J. C., Lord R. G., (1977). Implicit leadership theory: A potential threat to the internal validity of leader behavior questionnaires. Organizational Behavior and Human Performance, 20(1), 93–110. doi:10.1016/0030-5073(77)90046-0
49. Salgado J. F., Táuriz G., (2014). The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23(1), 3–30. doi:10.1080/1359432X.2012.716198
50. Samejima F., (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
51. Schmit M. J., Ryan A. M., (1993). The Big Five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78(6), 966–974. doi:10.1037/0021-9010.78.6.966
52. Scullen S. E., Mount M. K., Goff M., (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956. doi:10.1037/0021-9010.85.6.956
53. SHL. (1997). Inventory of Management Competencies: Manual and user’s guide. Surrey, UK. Saville & Holdsworth ltd.
54. Stark S., Chernyshenko O., Drasgow F., (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29, 184–203. doi:10.1177/0146621604273988
55. Stark S., Chernyshenko O. S., Drasgow F., (2011). Constructing fake-resistant personality tests using item response theory. In Ziegler M., MacCann C., Roberts R., (Eds.), New perspectives on faking in personality assessment (pp. 214–239). London: Oxford University Press.
56. Thorndike E. L., (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25–29. doi:10.1037/h0071663
57. Thurstone L. L., (1927). A law of comparative judgment. Psychological Review, 34, 273–286. doi:10.1037/h0070288
58. Timmerman M. E., Lorenzo-Seva U., (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209–220. doi:10.1037/a0023353
59. Van der Heijden B., Nijhof A., (2004). The value of subjectivity: Problems and prospects for 360-degree appraisal systems. The International Journal of Human Resource Management, 15(3), 493–511. doi:10.1080/0958519042000181223
60. Van Herk H., Poortinga Y. H., Verhallen T. M., (2004). Response styles in rating scales evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35(3), 346–360. doi:10.1177/0022022104264126
61. Warr P., Bourne A., (1999). Factors influencing two types of congruence in multirater judgments. Human Performance, 12(3-4), 183–210. doi:10.1080/08959289909539869
62. Warr P., Bourne A., (2000). Associations between rating content and self-other agreement in multi-source feedback. European Journal of Work and Organizational Psychology, 9(3), 321–334. doi:10.1080/135943200417948
63. Webster H., (1958). Correcting personality scales for response sets or suppression effects. Psychological Bulletin, 55(1), 62–64. doi:10.1037/h0048031
64. Williams L. J., Edwards J. R., Vandenberg R. J., (2003). Recent advances in causal modeling methods for organizational and management research. Journal of Management, 29(6), 903–936. doi:10.1016/S0149-2063_03_00084-9
65. Woehr D. J., Sheehan M. K., Bennett W., Jr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multirater approach. Journal of Applied Psychology, 90(3), 592–600. doi:10.1037/0021-9010.90.3.592
66. Yammarino F. J., (2003). Modern data analytic techniques for multisource feedback. Organizational Research Methods, 6(1), 6–14. doi:10.1177/1094428102239423