The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta-analysis

Organizational Research Methods

Volumn 10, Issue 4, 2007, Pages 689-709

  Bobko, Philip ^a   Roth, Philip L ^b   Buster, Maury A ^c  

^a GETTYSBURG COLLEGE   (United States)

^b CLEMSON UNIVERSITY   (United States)

^c Alabama State Personnel Department   (United States)

Author keywords

Composites; Content validity; Selection; Unit weights

Indexed keywords

EID: 34548542516     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428106294734     Document Type: Article

References (49)

1. * Aamodt, M., & Kimbrough, W. (1985). Comparison of four methods for weighting multiple predictors. Educational and Psychological Measurement, 45, 477-482.
2. * Arthur, W., Doverspike, D., & Barrett, G. (1996). Development of a job analysis-based procedure for weighting and combining content-related tests into a single test battery score. Personnel Psychology, 49, 971-985.
3. Biddle, D. (2005). Adverse impact and test validation: A practitioner's guide to valid and defensible employment testing. Burlington, VT: Gower.
4. Bobko, P. (2001). Correlation and regression (Rev. ed.). Thousand Oaks, CA: Sage.
5. Bobko, P., Roth, P., & Bobko, C. (2001). Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4, 46-61.
6. Bobko, P., & Schemmer, M. (1980). A note on standardized regression estimators. Psychological Bulletin, 88, 233-236.
7. Boreham, N., Russell, C., & Wastell, D. (1988). Can past academic criteria predict students at risk of future failure ? Medical Education, 22, 94-98.
8. Cascio, W. (1987). Costing human resources: The financial impact of behavior in organizations (2 nd ed.). Boston: PWS-Kent.
9. Claudy, J. (1972). A comparison of five variable weighting procedures. Educational and Psychological Measurement, 32, 311-322.
10. Clements, D., & Reid, R. (1994). Analytical MS/OR tools applied to a plant closure. Interfaces, 24, 1-12.
11. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 12, 1304-1312.
12. Cohen, J., West, S., Aiken, L., & Cohen, P. (2002). Applied multivariate regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
13. Dawes, R.M. (1974). The robust beauty of improper linear modes in decision making. American Psychologist, 34, 571-582.
14. Dawes, R.M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.
15. Dorans, N., & Drasgrow, F. (1978). Alternative weighting schemes for linear prediction. Organizational Behavior and Human Performance, 21, 316-345.
16. Doverspike, D., Winter, J., Healy, M., & Barrett, G. (1996). Simulations as a method of illustrating the impact of differential weights on personnel selection outcomes. Human Performance, 9, 259-273.
17. Einhorn, H., & Hogarth, R. (1975). Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13, 171-192.
18. * Fralicz, R., & Raju, N. (1982). A comparison of five methods for combining multiple criteria into a single composite. Educational and Psychological Measurement, 42, 823-827.
19. Gatewood, R., & Feild, H. (2001). Human resource selection (5 th ed.). New York: Harcourt.
20. * Goldstein, H., Rumison, K., & Yusko, K. (2002, April). The impact of test composite score weighting procedures on Black-White subgroup differences. Paper presented at the annual meeting of the Society for Industrial & Organizational Psychology, Toronto, Canada.
21. Goldstein, H., Yusko, K., Braverman, E., Smith, D., & Chung, B. (1998). The role of cognitive ability in the subgroup differences and incremental validity of assessment center exercises. Personnel Psychology, 51, 357-374.
22. Goldstein, I., Zedeck, S., & Schneider, B. (1993). An exploration of the job analysis-content validity process. In N. Schmitt & W. Borman (Eds.), Personnel selection in organizations (pp. 1-34). San Francisco: Jossey-Bass.
23. Grant, D., & Bray, D. (1970). Validation of employment tests for telephone company installation and repair occupations. Journal of Applied Psychology, 54, 7-14.
24. Guion, R. (1991). Personnel assessment, selection, and placement. In M. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2 nd ed., Vol. 2, pp. 327-397). Palo Alto, CA: Consulting Psychologists Press.
25. Hausknecht, J., Day, D., & Thomas, S. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57, 639-683.
26. Janz, T., Hellervik, L., & Gilmore, D. (1986). Behavior description interviewing: New, accurate, cost effective. Boston: Allyn & Bacon.
27. Laughlin, J. (1978). Comment on "estimating coefficients in linear models: It don't make no nevermind." Psychological Bulletin, 85, 247-253.
28. Lawshe, C. (1969). Statistical theory and practice in applied psychology. Personnel Psychology, 22, 117-123.
29. McCall, M., & Bobko, P. (1990). Research methods and discovery in industrial/organizational psychology. In M. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2 nd ed., Vol. 1, pp. 381-418). Palo Alto, CA: Consulting Psychologists Press.
30. Meehl, P. (1965). Seer over sign: The first good example. Journal of Experimental Research in Personality, 1, 27-32.
31. * O'Leary, R., O'Leary, B., & MacLane, C. (2004, April). Weighting content valid composites: A cautionary note on unit weights. Paper presented at the nineteenth annual meeting of the Society for Industrial & Organizational Psychology, Chicago.
32. Raju, N., Bilgic, R., Edwards, J., & Fleer, P. (1997). Methodology review: Estimation of population validity and cross validity, and the use of equal weights in prediction. Applied Psychological Measurement, 21, 291-305.
33. Raju, N., Bilgic, R., Edwards, J., & Fleer, P. (1999). Accuracy of population validity and cross-validity estimation: An empirical comparison of formula-based, traditional empirical, and equal weights procedures. Applied Psychological Measurement, 23, 99-115.
34. Ree, M., Carretta, T., & Earles, J. (1998). In top-down decisions, weighting variables does not matter: A consequence of Wilk's theorem. Organizational Research Methods, 1, 407-420.
35. Reynolds v. Alabama Department of Transportation, No. 85-665 (Memorandum Opinion & Order) (M. D. Ala. December 19, 2003 ).
36. Rosenberg, N., Skordahl, D., & Kaplan, H. (1961 a, May). Validation of fixed-wing aptitude battery against success in Army flight training. Human Factors Research Branch Technical Research Note 112, Adjutant General's Office (Army). Washington, DC.
37. Rosenberg, N., Skordahl, D., & Kaplan, H. (May, 1961 b). Validation of fixed-wing aptitude battery against success in ROTC flight training. Human Factors Research Branch Technical Research Note 111 ; Army Project 2L95-60-001, Adjutant General's Office (Army), Washington, DC.
38. Roth, P., & Switzer, F. (2002). Outliers and influential cases: Handling those discordant contaminated maverick rogues. In S. Rogelberg (Ed.), Handbook of methods in industrial and organizational psychology (pp. 297-309). Malden, MA: Blackwell.
39. Schmidt, F. (1971). The relative efficiency of regression and simple unit weight predictor weights in applied differential psychology. Educational and Psychological Measurement, 31, 699-714.
40. Schmidt, F., Hunter, J., & Urry, V. (1976). Statistical power in criterion-related validity studies. Journal of Applied Psychology, 61, 473-485.
41. Schmidt, F., & Le, H. (2004). Software for the Hunter-Schmidt meta-analysis methods. Iowa City: University of Iowa, Department of Management & Organization.
42. Seashore, H. (1957). Watch your weights. Test Service Bulletin, 52, 2-5.
43. Stillwell, W., Barron, F., & Edwards, W. (1983). Evaluating credit applications: A validation of multiattribute utility weight elicitation techniques. Organizational Behavior and Human Performance, 32, 87-108.
44. Trattner, M. (1963). Comparison of three methods for assembling aptitude test batteries. Personnel Psychology, 16, 221-232.
45. Wainer, H. (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin, 83, 213-217.
46. Wallace, M., & Schwab, D. (1976). A cross-validated comparison of five models used to predict graduate admissions committee decisions. Journal of Applied Psychology, 61, 559-563.
47. Wang, M., & Stanley, J. (1970). Differential weighting: A review of methods and empirical studies. Review of Educational Research, 40, 663-705.
48. Werts, C., & Linn, R. (1970). Path analysis: Psychological examples. Psychological Bulletin, 74, 193-212.
49. Wilks, S. (1938). Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika, 3, 23-40.