Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability

Journal of Applied Psychology

Volumn 92, Issue 2, 2007, Pages 373-385

  Hausknecht, John P ^a   Halpert, Jane A ^b   Di Paolo, Nicole T ^b   Moriarty Gerrard, Meghan O ^b  

^a Department of Obstetrics Gynecology   (United States)

^b DEPAUL UNIVERSITY   (United States)

Author keywords

Cognitive ability testing; Meta analysis; Personnel selection; Practice effects

Indexed keywords

ADULT; ARTICLE; CLINICAL TRIAL; COGNITION; HUMAN; INTELLIGENCE; LEARNING; META ANALYSIS; PERSONNEL MANAGEMENT; TEACHING;

ADULT; COGNITION; HUMANS; INTELLIGENCE; PERSONNEL SELECTION; PRACTICE (PSYCHOLOGY); TEACHING;

EID: 33947394098     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.92.2.373     Document Type: Article

References (86)

1. Allalouf, A., & Ben-Shakhar, G. (1998). The effect of coaching on the predictive validity of scholastic aptitude tests. Journal of Educational Measurement, 35, 31-47.
2. Anastasi, A. (1981). Coaching, test sophistication, and developed abilities. American Psychologist, 36, 1086-1093.
3. Anastasi, A., & Urbina, S. (1997). Psychological testing. Upper Saddle River, NJ: Prentice Hall.
4. Arvey, R. D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivational components of test taking. Personnel Psychology, 43, 695-716.
5. Babad, E. Y., & Bashi, J. (1977). Age and coaching effects on the reasoning performance of disadvantaged and advantaged Israeli children. Journal of Social Psychology, 103, 169-176.
6. Basso, M. R., Carona, F. D., Lowery, N., & Axelrod, B. N. (2002). Practice effects on the WAIS-III across 3- and 6-month intervals. The Clinical Neuropsychologist, 16, 57-63.
7. Best, J. B. (1999). Cognitive psychology (5th ed.). Belmont, CA: Brooks/ Cole.
8. Bobko, P. (2001). Correlation and regression: Applications for industrial/ organizational psychology and management (2nd ed.). Thousand Oaks, CA: Sage.
9. Brounstein, P. J., & Holahan, W. (1987). Patterns of change in Scholastic Aptitude Test performance among academically talented adolescents. Roeper Review, 10, 110-116.
10. Bunting, B. P., & Mooney, E. (2001). The effects of practice and coaching on test results for educational selection at eleven years of age. Educational Psychology, 21, 243-253.
11. Cahan, S., & Cohen, N. (1989). Age versus schooling effects on intelligence development. Child Development, 60, 1239-1249.
12. Campbell, D. T., & Kenny, D. A. (1999). A primer on regression artifacts. New York: Guilford Press.
13. Carretta, T. R., Zelenski, W. E., & Ree, M. J. (2000). Basic Attributes Test (BAT) retest performance. Military Psychology, 12, 221-232.
14. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor analytic studies. New York: Cambridge University Press.
15. Catron, D. W., & Thompson, C. C. (1979). Test-retest gains in WAIS scores after four retest intervals. Journal of Clinical Psychology, 35, 352-357.
16. Chernyshenko, O. S., & Ones, D. S. (1999). How selective are psychology graduate programs? The effects of the selection ratio on GRE scorevalidity. Educational and Psychological Measurement, 59, 951-961.
17. Cliffordson, C. (2004). Effects of practice and intellectual growth on performance on the Swedish Scholastic Aptitude Test (SweSAT). European Journal of Psychological Assessment, 20, 192-204.
18. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.
19. Colver, R. M., & Spielberger, C. D. (1961). Further evidence of a practice effect on the Miller Analogies Test. Journal of Applied Psychology, 45, 126-127.
20. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
21. Cortina, J. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415-439.
22. Dodrill, C. B. (1983). Long-term reliability of the Wonderlic Personnel Test. Journal of Consulting and Clinical Psychology, 51, 316-317.
23. Droege, R. C. (1966). Effects of practice on aptitude scores. Journal of Applied Psychology, 50, 306-310.
24. Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170-177.
25. Elwood, D. L., & Clark, C. L. (1978). Computer administration of the Peabody Picture Vocabulary Test to young children. Behavior Research Methods & Instrumentation, 10, 43-46.
26. Evans, F. R., & Pike, L. W. (1973). The effects of instruction for three mathematics item formats. Journal of Educational Measurement, 10, 257-272.
27. Ferrer, E., Salthouse, T. A., Steward, W. F., & Schwartz, B. S. (2004). Modeling age and retest processes in longitudinal studies of cognitive abilities. Psychology and Aging, 19, 243-259.
28. Frankel, E. (1960). Effects of growth, practice, and coaching on Scholastic Aptitude Test scores. Personnel and Guidance Journal, 38, 713-719.
29. Gavurin, E. I. (1973). Practice effect in anagram solving. Journal of Psychology, 84, 279-282.
30. Gilliland, S. W. (1993). The perceived fairness of selection systems: An organizational justice perspective. Academy of Management Review, 18, 694-734.
31. Gilmore, M. E. (1927). Coaching for intelligence tests. Journal of Educational Psychology, 18, 119-121.
32. Goodman, J. T., Streiner, D. L., & Woodward, C. A. (1974). Test-retest reliability of the Shipley Institute of Living scale: Practice effects or random variation. Psychological Reports, 35, 351-354.
33. Gordon, M. E., Slade, L. A., & Schmitt, N. (1986). The "science of the sophomore" revisited: From conjecture to empiricism. Academy of Management Review, 11, 191-207.
34. Hager, W., & Hasselhorn, M. (1997). Wirkungen der testwiederholung und entwicklungsbedingte leistungssteigerungen bei der durchfuhrung des CFT 1 mit erstklässlern [Retest effects and effects of developmental changes on the CFT 1 score in testing first grade children]. Zeitschrift für Psychologie, 205, 205-229.
35. Harcourt Assessment, Inc. (2005). Miller Analogies Test: Scoring and score reporting. Retrieved June 5, 2005. from http://www.milleranalogies.com
36. Hausknecht, J. P., Halpert, J. A., Harder, B. K., Kuljanin, G., & Moriarty, M. (2005, April). Issues in repeated testing: Test attitudes and applicant reactions. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Los Angeles, CA.
37. Hausknecht, J. P., & Howard, M. J. (2004, August). Effects of candidate retesling in an employment context. Paper presented at the meeting of the Academy of Management, New Orleans, LA.
38. Hausknecht, J. P., Trevor, C. O., & Farr, J. L. (2002). Retaking ability tests in a selection setting: Implications for practice effects, training performance, and turnover. Journal of Applied Psychology, 87, 243-254.
39. Holloway, H. D. (1954). Effects of training on the SRA Primary Mental Abilities (primary) and the WISC. Child Development, 25, 253-263.
40. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
41. Koenig, J. A., & Leger, K. F. (1997). Test-taking behaviors and their impact on performance. Academic Medicine, 72, S100-S103.
42. Kreit, L. H. (1968). The effects of test-taking practice on pupil test performance. American Educational Research Journal, 5, 616-625.
43. Kulik, J. A., Bangert-Drowns, R. L., & Kulik, C. C. (1984). Effectiveness of coaching for aptitude tests. Psychological Bulletin, 95, 179-188.
44. Kulik, J. A., Kulik, C. C., & Bangert, R. L. (1984). Effects of practice on aptitude and achievement test scores. American Educational Research Journal, 21, 435-447.
45. Lane, R. G., Penn, N. E., & Fischer, R. F. (1966). Miller Analogies Test: A note on permissive retesting. Journal of Applied Psychology, 50, 409-411.
46. Lievens, F., Buyse, T., & Sackett, P. R. (2005). Relest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981-1007.
47. Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: The good, bad, and ugly. The Annals of the American Academy of Political and Social Science, 587, 69-81.
48. McIntyre, W. (1954). Difficulty of understanding instructions as a factor in coaching and practice effects in intelligence testing. British Journal of Educational Psychology, 24, 122-123.
49. McRae, H. (1942). The inconstancy of group test I. Q.'s. British Journal of Educational Psychology, 12, 59-70.
50. Merriman, C. (1927). Coaching for mental tests. Educational Administration and Supervision, 13, 59-64.
51. Messick, S., & Jungeblut, A. (1981). Time and method in coaching for the SAT. Psychological Bulletin, 89, 191-216.
52. Nathan, J. S., & Camara, W. J. (1998, September). Score change when retaking the SAT I: Reasoning Test (Research Note No. RN-05). New York: The College Board.
53. Nkaya, I. N., Huteau, M., & Bonnet, J.-P. (1994). Retest effect on cognitive performance on the Raven-38 matrices in France and in the Congo. Perceptual and Motor Skills, 78, 503-510.
54. Oakland, T. (1972). The effects of test-wiseness materials on standardized test performance of preschool disadvantaged children. Journal of School Psychology, 10, 355-360.
55. Powell, B., & Steelman, L. C. (1983). Equity and the LSAT. Horvard Educational Review, 53, 32-44.
56. Powers, D. E. (1985). The effects of test preparation on the validity of a graduate admission test. Applied Psychological Measurement, 9, 179-190.
57. Powers, D. B. (1986). Relations of test item characteristics to test preparation/test practice effects: A quantitative summary. Psychological Bulletin, 100, 67-77.
58. Powers, D. E., & Rock, D. A. (1999). Effects of coaching on SAT I: Reasoning test scores. Journal of Educational Measurement, 36, 93-118.
59. Quereshi, M. Y. (1968). Practice effects on the WISC subtest scores and IQ estimates. Journal of Clinical Psychology, 24, 79-85.
60. Reeve, C. L., & Lam, H. (2004, April). The relation between practice effects, scale properties, and test-taker characteristics. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Chicago, IL.
61. Richardson, F., & Robinson, E. S. (1921). Effects of practice upon the scores and predictive value of the Alpha Intelligence Examination. Journal of Experimental Psychology, 4, 300-317.
62. Roth, P. I., Bevier, C. A., Bobko, P., Switzer, F. S., III, & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330.
63. Ruiz, R. A., & Krauss, H. H. (1967). Test-retest reliability and practice effect with the Shipley Institute of Living Scale. Psychological Reports, 20, 1085-1086.
64. Ryan, J. J., Morris, J., Yaffa, S., & Peterson, L. (1981). Test-retest reliability of the Wechsler Memory Scale, Form I. Journal of Clinical Psychology, 37, 847-848.
65. Sackett, P. R., Burris, L. R., & Ryan, A. M. (1989). Coaching and practice effects in personnel selection. In C. Cooper & I. Robertson (Eds.), International review of industrial and organizational psychology (pp. 145-183). New York: Wiley.
66. Sanchez, R. J., Truxillo, D. M., & Bauer, T. N. (2000). Development and examination of an expectancy-based measure of test-taking motivation. Journal of Applied Psychology, 85, 739-750.
67. Schubert, J. (1967). Effect of training on the performance of the W.I.S.C. "block design" subtest. British Journal of Social and Clinical Psychology, 6, 144-149.
68. Scullen, S. E., Bergey, P. K., & Aiman-Smith, L. (2005). Forced distribution rating systems and the improvement of workforce potential: A baseline simulation. Personnel Psychology, 58, 1-32.
69. Shany, M. T., & Biemiller, A. (1995). Assisted reading practice: Effects on performance for poor readers in grades 3 and 4. Reading Research Quarterly, 30, 382-395.
70. Sin, H. P., Farr, J. L., Murphy, K. R., & Hausknecht, J. P. (2004, August). An investigation of Black-White differences in self-selection and performance in repeated testing. Paper presented at the meeting of the Academy of Management, New Orleans, LA.
71. Smither, J. W., London, M., & Reilly, R. R. (2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology, 58, 33-66.
72. Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). College Park, MD: Author.
73. Spielberger, C. D. (1959). Evidence of a practice effect on the Miller Analogies Test. Journal of Applied Psychology, 43, 259-263.
74. Steel, P. D., & Kammeyer-Mueller, J. D. (2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87, 96-111.
75. Stricker, L. J. (1984). Test disclosure and retest performance on the SAT. Applied Psychological Measurement, 8, 81-87.
76. Te Nijenhuis, J., Voskuijl, O. F., & Schijve, N. B. (2001). Practice and coaching on IQ tests: Quite a lot of g. International Journal of Selection and Assessment, 9, 302-308.
77. Thomas, P. H., Busciglio, H. H., & Goldenberg, R. J. (2004, April). An investigation of the effects of applicant retesting on assessment effectiveness: A look at practical implications within U.S. Customs and Border Protection. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Chicago, IL.
78. Tuma, J. M., Appelbaum, A. S., & Bee, D. E. (1978). Comparability of the WISC and the WISC-R in normal children of divergent socioeconomic backgrounds. Psychology in the Schools, 15, 339-346.
79. Tuzinski, K. A., Laczo, R. M., & Sackett, P. R. (2005, April). Impact of response distortion on retaking of cognitive and personality tests. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Los Angeles, CA.
80. U.S. Department of Labor. (1999). Testing and assessment: An employer's guide to good practices. Washington, DC: Author, Employment and Training Administration.
81. van der Reis, A. P. (1963). Is retesting justified in personnel selection? Psychologia Africana, 10, 19-30.
82. Vernon, P. E. (1961). The structure of human abilities. London: Methuen.
83. Vroom, V. H. (1964). Work and motivation. Oxford, England: Wiley.
84. Whitely, S. E., & Dawis, R. E. (1974). Effects of cognitive intervention on latent ability measured from analogy items. Journal of Educational Psychology, 66, 710-717.
85. Wing, H. (1980). Practice effects with traditional mental test items. Applied Psychological Measurement, 4, 141-155.
86. Woehlke, A. B., & Wilder, D. H. (1963). Differences in difficulty of Forms A and B of the Otis Self-Administering Test of Mental Ability. Personnel Psychology, 16, 395-398.