Test Length and Decision Quality in Personnel Selection: When Is Short Too Short?

International Journal of Testing

Volumn 12, Issue 4, 2012, Pages 321-344

  Kruyen, Peter M ^a   Emons, Wilco H M ^b   Sijtsma, Klaas ^b  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b TILBURG UNIVERSITY   (Netherlands)

Author keywords

decision quality; personnel selection; test length reduction

Indexed keywords

EID: 84867965783     PISSN: 15305058     EISSN: 15327574     Source Type: Journal    
DOI: 10.1080/15305058.2011.643517     Document Type: Article

References (61)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. 1985. Standards for educational and psychological testing, Washington, DC: American Educational Research Association.
2. Arthur, W. and Day, D. V. 1994. Development of a short-form for the Raven Advanced Progressive Matrices. Educational and Psychological Measurement, 54: 394-403. doi: 10.1177/ 0013164494054002013
3. Arvey, R. D. and Renz, G. L. 1992. Fairness in the selection of employees. Journal of Business Ethics, 11: 331-340. doi: 10.1007/BF00870545
4. Baron, H. and Austin, J. 2000. Measuring ability via the Internet: Opportunities and issues Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA
5. Bartram, D. and Hambleton, R. K. 2006. Computer-based testing and the internet: Issues and advances, West Sussex, UK: John Wiley & Sons.
6. Berk, R. A. 1986. A consumer's guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56: 137-172. doi: 10.3102/00346543056001137
7. Born, M. and Scholarios, D. 2005. "Decision making in selection". In The Blackwell handbook of selection, Edited by: Evers, A., Anderson, N. and Smit-Voskuijl, O. O. 267-290. Malden, MA: Blackwell Publishing.
8. Borsboom, D., Romeijn, J. and Wicherts, J. M. 2008. Measurement invariance versus selection invariance: Is fair selection possible?. Psychological Methods, 13: 75-98. doi: 10.1037/1082-989X.13.2.75
9. Brown, W. 1910. Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3: 296-322. doi: 10.1111/j.2044-8295.1910.tb00207.x
10. Burisch, M. 1997. Test length and validity revisited. European Journal of Personality, 11: 303-315. doi: 10.1002/(SICI)1099-0984(199711)11:4<303::AID-PER292>3.0.CO;2-#
11. Caldwell, D. F. and O'Reilly, C. A. III. 1990. Measuring person-job fit with a profile-comparison process. Journal of Applied Psychology, 75: 648-657. doi: 10.1037/0021-9010.75.6.648
12. Campbell, J. P. 1979. "Problems and new directions for industrial psychology". In Handbook of industrial and organizational psychology, Edited by: Dunnette, M. D. 185-222. Chicago, IL: Rand McNally College Publishing Company.
13. Candell, G. L. and Drasgow, F. 1988. An iterative procedure for linking metrics and assessing item bias in item response theory. Journal of Psychological Measurement, 12: 253-260. doi: 10.1177/014662168801200304
14. Cascio, W. F. 1980. Responding to the demand for accountability: A critical analysis of three utility models. Organizational Behavior and Human Performance, 25: 32-45. doi: 10.1016/0030-5073(80)90024-0
15. Charter, R. A. 2003. A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability. The Journal of General Psychology, 130: 290-304. doi: 10.1080/00221300309601160
16. Cohen, A. S., Kim, S. H. and Baker, F. B. 1993. Detection of differential item functioning in the Graded Response Model. Applied Psychological Measurement, 17: 335-350. doi: 10.1177/014662169301700402
17. Cook, M. 2009. Personnel selection: Adding value through people, (5th ed.), London, UK: Wiley-Blackwell.
18. Costa, P. T. Jr. and McCrae, R. R. 1992. The NEO PI/FFI manual supplement, Odessa, FL: Psychological Assessment Resources.
19. Cronbach, L. J. and Gleser, G. C. 1953. Assessing similarity profiles. Psychological Bulletin, 50: 456-473.
20. Crowder, B. and Michael, W. B. 1991. The development and validation of a short form of a multidimensional self-concept measure for high technology employees. Educational and Psychological Measurement, 51: 447-454. doi: 10.1177/0013164491512019
21. Dalen, L. H., Stanton, N. A. and Roberts, A. D. 2001. Faking personality questionnaires in personnel selection. The Journal of Management Development, 20: 729-741. doi: 10.1108/02621710110401428
22. Dilchert, S. 2007. Peaks and valleys: Predicting interests in leadership and managerial positions from personality profiles. International Journal of Selection and Assessment, 15: 317-334. doi: 10.1111/j.1468-2389.2007.00391.x
23. Drasgow, F., Nye, C. D. and Tay, L. 2010. "Indicators of quality assessment". In Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent, Edited by: Scott, J. C. and Reynolds, D. H. 27-60. San Francisco, CA: Jossey-Bass.
24. Edwards, J. R. 1993. Problems with the use of profile similarity indexes in the study of congruence in organizational research. Personnel Psychology, 46: 641-665. doi: 10.1111/j.1744-6570.1993.tb00889.x
25. Embretson, S. E. and Reise, S. P. 2000. Item response theory for psychologists, Mahwah, NJ: Lawrence Erlbaum Associates.
26. Emons, W. H. M., Sijtsma, K. and Meijer, R. R. 2007. On the consistency of individual classification using short scales. Psychological Methods, 12: 105-120. doi: 10.1037/1082-989x.12.1.105
27. Ercikan, K. and Julian, M. 2002. Classification accuracy of assigning student performance to proficiency levels: Guidelines for assessment design. Applied Measurement in Education, 15: 269-294. doi: 10.1207/S15324818AME1503_3
28. Forsythe, C., Grose, E. and Ratner, J. 1998. Human factors and Web development, Mahwah, NJ: Lawrence Erlbaum Associates.
29. Fraley, R. C., Waller, N. G. and Brennan, K. A. 2000. An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78: 350-365. doi: 10.1037/0022-3514.78.2.350
30. Gardner, D. G., Cummings, L. L., Dunham, R. B. and Pierce, J. L. 1998. Single-item versus multiple-item measurement scales: An empirical comparison. Educational and Psychological Measurement, 58: 898-915. doi: 10.1177/0013164498058006003
31. Gardner, D. G., Dunham, R. B., Cummings, L. L. and Pierce, J. L. 1989. Focus of attention at work: Construct definition and empirical validation. Journal of Occupational Psycholog, 62: 61-77.
32. Glaros, A. G. and Kline, R. B. 1988. Understanding the accuracy of tests with cutting scores: The sensitivity, specificity, and predictive value model. Journal of Clinical Psychology, 44: 1013-1023. doi: 10.1002/1097-4679(198811)44:6<1013::AID-JCLP2270440627>3.0.CO;2-Z
33. Gosling, S. D., Rentfrow, P. J. and Swann, W. B. Jr. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37: 504-528. doi: 10.1016/S0092-6566(03)00046-1
34. Guion, R. M. 1976. "Recruiting, selection, and job placement". In Handbook of industrial and organizational psychology, Edited by: Dunnette, M. D. 777-828. Chicago, IL: Rand McNally College Publishing.
35. Hambleton, R. K. and Slater, S. C. 1997. Reliability of credentialing examinations and the impact of scoring models and standard-setting policies. Applied Measurement in Education, 10: 19-38. doi: 10.1207/s15324818ame1001_2
36. Harwell, M. R. and Janosky, J. E. 1991. An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15: 279-291. doi: 10.1177/014662169101500308
37. Judge, T. A., Erez, A., Bono, J. E. and Thoresen, C. J. 2003. The core self-evaluations scale: Development of a measure. Personnel Psychology, 56: 303-331. doi: 10.1111/j.1744-6570.2003.tb00152.x
38. Kline, P. 2000. The handbook of psychological testing, (2nd ed.), New York, NY: Routledge.
39. Konradt, U., Hertel, G. and Joder, K. 2003. Web-based assessment of call center agents: Development and validation of a computerized instrument. International Journal of Selection and Assessment, 11: 184-193. doi: 10.1111/1468-2389.00242
40. Levy, P. 1968. Short-form tests: A methodological review. Psychological Bulletin, 69: 410-416. doi: 10.1037/h0025736
41. Lord, F. M. and Novick, M. R. 1968. Statistical theories of mental test scores, Reading, MA: Addison-Wesley.
42. Lord, F. M. and Wingersky, M. S. 1984. Comparison of IRT true-score and equipercentile observed-score equatings. Applied Psychological Measurement, 8: 453-461. doi: 10.1177/ 014662168400800409
43. Martin, D. E., Moore, C. F., Sloan, L. and Legree, P. J. 2007. Personnel selection: An application of the unobtrusive knowledge test. Journal of Business and Behavioral Sciences, 16: 4-16. Retrieved fromhttp://ssrn.com/abstract=1739396
44. Meehl, P. E. and Rosen, A. 1955. Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52: 194-216. doi: 10.1037/h0048070
45. Mosel, J. N. 1953. Single-item tests for personnel screening. Educational and Psychological Measurement, 13: 179-192. doi: 10.1177/001316445301300204
46. Murphy, K. R. 1994. Potential effects of banding as a function of test reliability. Personnel Psychology, 47: 477-495. doi: 10.1111/j.1744-6570.1994.tb01733.x
47. Netemeyer, R. G., Pulling, C. and Bearden, W. O. 2003. "Observations on some key psychometric properties of paper-and-pencil measures". In Advances in business marketing and purchasing: Essays by distinguished scholars, Edited by: Woodside, A. G. and Moore, E. M. 115-138. New York, NY: Elsevier Science.
48. Reise, S. P. and Waller, N. G. 2009. Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5: 27-48. doi: 10.1146/annurev.clinpsy.032408.153553
49. Samejima, F. 1969. Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17), Richmond, VA: Psychometric Society. Retrieved fromhttp://www.psychometrika.org/journal/online/MN17.pdf
50. Schmidt, F. L. and Hunter, J. E. 1998. The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124: 262-274. doi: 10.1037/0033-2909.124.2.262
51. Schmidt, F. L., Hunter, J. E., McKenzie, R. C. and Muldrow, T. W. 1979. Impact of valid selection procedures on work-force productivity. Journal of Applied Psychology, 64: 609-626. doi: 10.1037/0021-9010.64.6.609
52. Segall, D. O. 2001. General ability measurement: An application of multidimensional item response theory. Psychometrika, 66: 79-97. doi: 10.1007/BF02295734
53. Smith, G. T., McCarthy, D. M. and Anderson, K. 2000. On the sins of short-form development. Psychological Assessment, 12: 102-111. doi: 10.1037/1040-3590.12.1.102
54. Smith, M. and Smith, P. C. 2005. Testing people at work: Competencies in psychometric testing, Malden, MA: BPS Blackwell.
55. Smits, N. and Vorst, H. C. M. 2007. Reducing the length of questionnaires through structurally incomplete designs: An illustration. Learning and Individual Differences, 17: 25-34. doi: 10.1016/j.lindif.2006.12.005
56. Spearman, C. 1910. Correlation calculated from faulty data. British Journal of Psychology, 3: 271-295. doi: 10.1111/j.2044-8295.1910.tb00206.x
57. Stanton, J. M., Sinar, E. F., Balzer, W. K. and Smith, P. C. 2002. Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55: 167-194. doi: 10.1111/j.1744-6570.2002.tb00108.x
58. Taylor, H. C. and Russell, J. T. 1939. The relationship of validity coefficients to the practical effectiveness of tests in selection: Discussion and tables. Journal of Applied Psychology, 23: 565-578. doi: 10.1037/h0057079
59. Thissen, D., Pommerich, M., Billeaud, K. and Williams, V. S. L. 1995. Item response theory for scores including polytomous items with ordered responses. Applied Psychological Measurement, 19: 39-49. doi: 10.1177/014662169501900105
60. Van Iddekinge, C. H. and Ployhart, R. E. 2008. Developments in the criterion-related validation of selection procedures: A critical review and recommendations for practice. Personnel Psychology, 61: 871-925. doi: DOI 10.1111/j.1744-6570.2008.00133.x
61. Wanous, J. P. and Hudy, M. J. 2001. Single-item reliability: A replication and extension. Organizational Research Methods, 4: 361-375. doi: 10.1177/109442810144003