Quantitative differences in retest effects across different methods used to construct alternate test forms

Intelligence

Volumn 41, Issue 3, 2013, Pages 181-192

  Arendasy, Martin E ^a   Sommer, Markus ^a  

^a UNIVERSITY OF GRAZ   (Austria)

Author keywords

Automatic item generation; Cognitive ability; Identical vs. alternate test forms; Retest effect

Indexed keywords

EID: 84875766490     PISSN: 01602896     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.intell.2013.02.004     Document Type: Article

References (60)

1. Anastasi A. Coaching, test sophistication, and developed abilities. American Psychologist 1981, 36:1086-1093.
2. Andersen E.B. A goodness of fit test for the Rasch model. Psychometrica 1973, 8:123-140.
3. Arbuckle J.L. Amos 5.0 update to the Amos user's guide 2003, Smallwaters, Chicago.
4. Arendasy M., Sommer M. Automatic generation of quantitative reasoning items: A schema-based isomorphic approach. Learning and Individual Differences 2007, 17:366-383.
5. Arendasy M., Sommer M. Evaluating the contribution of different item features to the effect size of the gender difference in three-dimensional mental rotation using automatic item generation. Intelligence 2010, 38:574-581.
6. Arendasy M., Sommer M. Automatisierte Itemgenerierung: Aktuelle Ansätze, Anwendungen und Forschungen. Enzyklopädie für Psychologie: Methoden der Psychologischen Diagnostik 2011, 215-280. Hogrefe, Göttingen. L.F. Hornke, M. Amelang, M. Kersting (Eds.).
7. Arendasy M., Sommer M. Using automatic item generation to meet the increasing item demands of high-stakes assessment. Learning and Individual Differences 2012, 22:112-117.
8. Arendasy M., Sommer M., Gittler G., Hergovich A. Automatic generation of Quantitative Reasoning items: A pilot study. Journal of Individual Differences 2006, 27:2-14.
9. Arendasy M., Sommer M., Mayr F. Using automatic item generation to simultaneously construct German and English versions of a word fluency test. Cross Cultural Psychology 2012, 43:464-479.
10. Barry C.L., Horst S.J., Finney S.J., Brown A.R., Kopp J.P. Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing 2010, 10:342-363.
11. Bejar I.I. Generative testing: From conception to implementation. Item Generation for Test Development 2002, 199-218. Lawrence Erlbaum Associates, New Jersey. S.H. Irvine, P.C. Kyllonen (Eds.).
12. Bors D.A., Vigneau F. The effect of practice on Raven's Advanced Progressive Matrices. Learning and Individual Differences 2003, 13:291-312.
13. Carroll J.B. Human cognitive abilities: A survey of factor-analytic studies 1993, Cambridge University Press, New York.
14. Cheung G.W., Rensvold R.B. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling 2002, 9:233-255.
15. Drasgow F. Study of measurement bias of two standardized psychological tests. Journal of Applied Psychology 1987, 72:19-29.
16. Fischer G.H. Linear logistic models for change. Rasch models. Foundations, recent developments, and applications 1995, 157-180. Springer, New York. G.H. Fischer, I.W. Molenaar (Eds.).
17. Fischer G.H., Ponocny-Seliger E. LPCM-Win software and manual 1999, Assessment Software Corporation, MN.
18. Freund A.P., Hofer S., Holling H. Explaining and controlling for the psychometric properties of computer-generated figural matrix items. Applied Psychological Measurement 2008, 32:195-210.
19. Freund P.A., Holling H. How to get real smart: Modeling retest and training effects in ability testing using computer-generated figural matrices items. Intelligence 2011, 39:233-243.
20. Gittler G., Arendasy M. Endlosschleifen: Psychometrische Grundlagen des Aufgabentyps EP. Diagnostica 2003, 49:164-175.
21. Glas C.A., van der Linden W.J. Computerized adaptive testing with item cloning. Applied Psychological Measurement 2003, 27:247-261.
22. Hausknecht J.P., Halpert J.A., Di Paolo N.T., Moriarty Gerrard M.O. Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology 2007, 92:373-385.
23. Hausknecht J.P., Trevor C.O., Farr J.L. Retaking ability tests in a selection setting: Implications for practice effects, training performance, and turnover. Journal of Applied Psychology 2002, 87:243-254.
24. Heil M., Rösler F., Link M., Bajric J. What is improved if mental rotation task is repeated: The efficiency of memory access, or the speed of transformation routine?. Psychological Research 1998, 61:99-106.
25. Hu L.T., Bentler P.M. Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 1999, 6:1-55.
26. Irvine S.H. The foundations of item generation for mass testing. Item generation for test development 2002, 3-34. Lawrence Erlbaum Associates, Mahwah, NJ. S.H. Irvine, P.C. Kyllonnen (Eds.).
27. Irvine S.H., Kyllonen P.C. Item generation for test development 2002, 199-218. Lawrence Erlbaum Associates, New Jersey.
28. Jensen A.R. The g factor: The science of mental ability 1998, Prager, Westport.
29. Kulik J.A., Kulik C.-L.C., Bangert R.L. Effects of practice on aptitude and achievement test scores. American Educational Research Journal 1984, 21:435-447.
30. Lievens F., Buyse T., Sackett P.R. Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology 2005, 58:981-1007.
31. Lievens F., Reeve Ch.L., Heggestad E.D. An examination of psychometric bias due to retesting on cognitive ability tests in selection settings. Journal of Applied Psychology 2007, 92:1672-1682.
32. Lubke G.H., Dolan C.V., Kelderman H., Mellenbergh G.J. On the relationship between sources of within- and between-group differences and measurement invariance in common factor model. Intelligence 2003, 31:543-566.
33. Major J.T., Johnson W., Bouchard T.J. The dependability of the general factor of intelligence: Why small, single-factor models do not adequately represent g. Intelligence 2011, 39:418-433.
34. Marsh H.W., Hau K.T., Wen Z. In search of golden rules: Comment on hypothesis testing approaches to setting cut off values for fit indexes and dangers in overgeneralising Hu and Bentler's (1999) findings. Structural Equation Modeling 2004, 11:320-341.
35. Matton N., Vautier St, Raufaste E. Situational effects may account for gain scores in cognitive ability testing: A longitudinal SEM approach. Intelligence 2009, 37:412-421.
36. Matton N., Vautier St, Raufaste E. Test-specificity of the advantage of retaking cognitive ability tests. International Journal of Selection and Assessment 2011, 19:11-17.
37. Meade A.W., Lautenschlager G.J., Hecht J.E. Establishing measurement equivalence and invariance in longitudinal data with item response theory. International Journal of Testing 2005, 5:279-300.
38. Millsap R.E. Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives 2010, 4:5-9.
39. Morley M.E., Bridgeman B., Lawless R.R. Transfer between variants of quantitative items. GRE Board Rep. No. 00-06R 2004, ETS, Princeton.
40. Penfield R.D., Camilli G. Differential item functioning and item bias. Handbook of statistics vol. 26 - Psychometrics 2007, 125-167. Elsevier, North Holland. C.R. Rao, S. Sinharay (Eds.).
41. Raju N.S., Laffine L.J., Byrne B.M. Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology 2002, 87:517-529.
42. Rapport L.J., Brines D.B., Axelrod B.N., Theisen M.E. Full scale IQ as mediator of practice effects: The rich get richer. The Clinical Neuropsychologist 1997, 11:375-380.
43. Rasch G. Probabilistic models for some intelligence and attainment tests 1980, The University of Chicago Press, Chicago.
44. Reeve Ch.L., Bonaccio S. Does test anxiety induce measurement bias in cognitive ability tests?. Intelligence 2008, 36:526-538.
45. Reeve Ch.L., Heggestad E.D., Lievens F. Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence 2009, 37:34-41.
46. Reeve Ch.L., Lam H. The psychometric paradox of practice effects due to retesting: Measurement invariance and stable ability estimates in the face of observed score changes. Intelligence 2005, 33:535-549.
47. Reeve Ch.L., Lam H. The relation between practice effects, test-taker characteristics and degree of g-factor saturation. International Journal of Testing 2007, 7:225-242.
48. Reeve Ch.L., Lam H. Consideration of g as a common antecedent for cognitive ability test performance, test motivation, and perceived fairness. Intelligence 2007, 35:347-358.
49. Salthouse T.A., Tucker-Drob E.M. Implications of short-term retest effects for the interpretation of longitudinal change. Neuropsychology 2008, 22:800-811.
50. Suárez-Falcon J.C., Glas C.A.W. Evaluation of global testing procedures for item fit to the Rasch model. The British Journal of Mathematical and Statistical Psychology 2003, 56:127-143.
51. Tanzer N.K. Developing tests for use in multiple languages and cultures: A plea for simultaneous development. Adapting educational and psychological tests for cross-cultural assessment 2005, 235-264. Lawrence Erlbaum Associates, New Jersey. R.K. Hambleton, P.F. Merenda, C. Spielberger (Eds.).
52. te Nijenhuis J., Vianen A.E.M., van der Flier H. Score gains on g-loaded tests: No g. Intelligence 2007, 35:283-300.
53. Te Nijenhuis J., Voskuijl O.F., Schijve N.B. Practice and coaching on IQ tests: Quite a lot of g. International Journal of Selection and Assessment 2001, 9:302-308.
54. Theisen M.E., Rapport L.J., Axelrod B.N., Brines D.B. Effects of practice in repeated administrations of the Wechsler Memory Scale-Revised in normal adults. Psychological Assessment 1998, 5:85-92.
55. van de Vijver F.J.R. Inductive reasoning in Zambia, Turkey, and the Netherlands: Establishing cross-cultural equivalence. Intelligence 2002, 30:313-351.
56. van Iddekinge Ch.H., Morgeson F.P., Schleicher D.J., Campion M.A. Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting. Journal of Applied Psychology 2011, 96:941-966.
57. Verguts T., De Boeck P. A note on the Martin-Löf test for unidimensionality. Methods of Psychological Research - Online 2000, 5:77-82.
58. Wickerts J.M., Scholten A.Z. Test anxiety and the validity of cognitive tests: A confirmatory factor analyses perspective and some empirical findings. Intelligence 2010, 38:169-178.
59. Wu A.D., Li Z., Zumbo B.D. Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research and Evaluation 2007, 12:1-26.
60. Zeidner M. Test anxiety: The state of the art 1998, Plenum, New York.