How to get really smart: Modeling retest and training effects in ability testing using computer-generated figural matrix items

Intelligence

Volumn 39, Issue 4, 2011, Pages 233-243

  Freund, Philipp Alexander ^a   Holling, Heinz ^b  

^a LEUPHANA UNIVERSITY OF LÜNEBURG   (Germany)

^b UNIVERSITY OF MÜNSTER   (Germany)

Author keywords

Figural matrix items; Individual change; Rational item construction; Retest effects; Training effects

Indexed keywords

EID: 79957661124     PISSN: 01602896     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.intell.2011.02.009     Document Type: Article

References (62)

1. AERA, APA, NCME Standards for educational and psychological testing 1999, APA, Washington, DC.
2. Anastasi A. Coaching, test sophistication, and developed abilities. American Psychologist 1981, 36:1086-1093.
3. Bejar I.I. Generative testing: From conception to implementation. Item generation for test development 2002, 199-217. Lawrence Erlbaum Associates, Mahwah, NJ. S.I. Irvine, P.C. Kyllonen (Eds.).
4. Bors D.A., Vigneau F. The effect of practice on Raven's advanced progressive matrices. Learning and Individual Differences 2003, 13:291-312.
5. Carpenter P.A., Just M.A., Shell P. What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychological Review 1990, 97:404-431.
6. Carroll J.B. Human cognitive abilities 1993, Cambridge University Press, Cambridge.
7. Cattell R.B. Are culture fair intelligence tests possible and necessary?. Journal of Research and Development in Education 1979, 12:2-13.
8. Christian M.S., Edwards B.D., Bradley J.C. Situational judgments tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology 2010, 63:83-117.
9. Corman L., Budoff M. Factor structures of Spanish-speaking and non-Spanish-speaking children on Raven's progressive matrices. Educational and Psychological Measurement 1974, 34:977-981.
10. Coyle T.R. Test-retest changes on scholastic aptitude tests are not related to g. Intelligence 2006, 34:15-27.
11. Denney N.W., Heidrich S.M. Training effects on Raven's progressive matrices in young, middle-aged, and elderly adults. Psychology and Aging 1990, 5:144-145.
12. Diemand A., Schuler H., Stapf K.H. Zum Einsatz eines Lerntests bei Ingenieurstudenten - Eine Pilotstudie [Introduction of a trainability test for engineering students: A pilot study]. Zeitschrift für Arbeits- und Organisationspsychologie 1991, 35:15-22.
13. Dunlap K., Snyder A. Practice effects in intelligence tests. Journal of Experimental Psychology 1920, 3:396-403.
14. Embretson S.E. A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods 1998, 3:380-396.
15. Embretson S.E. Generating abstract reasoning items with cognitive theory. Item generation for test development 2002, 219-250. Lawrence Erlbaum Associates, Mahwah, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
16. Fischer G.H. The linear logistic test model as an instrument in educational research. Acta Psychologica 1973, 37:359-374.
17. Fischer G.H. Linear logistic models for change. Rasch models: Foundations, recent developments, and applications 1995, 157-180. Springer, New York. G. Fischer, I. Molenaar (Eds.).
18. Freund Ph.A., Hofer S., Holling H. Explaining and controlling for the psychometric properties of computer-generated figural matrix items. Applied Psychological Measurement 2008, 32:195-210.
19. Frey M.C., Detterman D.K. Scholastic assessment or g? The relationship between the scholastic assessment test and general cognitive ability. Psychological Science 2004, 15:373-378.
20. Hausknecht J.P., Halpert J.A., Di Paolo N.T., Moriarty Gerrard M.O. Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology 2007, 92:373-385.
21. Hausknecht J.P., Trevor C.O., Farr J.L. Retaking ability tests in a selection setting: Implications for practice effects, training performance, and turnover. Journal of Applied Psychology 2002, 87:243-254.
22. Hesse F.W. Training-induced changes in problem solving. Zeitschrift für Psychologie 1982, 190:405-423.
23. Hofer, S. (2004). MatrixDeveloper [unpublished computer software]. Münster, Germany: Psychological Institute, Westfälische Wilhelms-Universität.
24. Jaeggi S.M., Buschkuehl M., Jonides J., Perrig W.J. Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America 2008, 105:6829-6833.
25. Jensen A.R. The g factor: The science of mental ability 1998, Praeger, Westport, CT.
26. Johnson C., Raudenbush S.W. A repeated measures, multilevel Rasch model with application to self-reported criminal behavior. Methodological issues in aging research 2006, 131-164. Lawrence Erlbaum Associates, Mahwah, New Jersey. C.S. Bergeman, S.M. Boker (Eds.).
27. Klauer K.J. "Bottom up" oder "top down"? Über die Transferwirkungen zweier Strategien zum Training des induktiven Denkens [Bottom up or top down? On the transfer of two strategies for training inductive reasoning]. Sprache und Kognition 1992, 11:91-103.
28. Klauer K.J., Willmes K., Phye G.D. Inducing inductive reasoning: Does it transfer to fluid intelligence?. Contemporary Educational Psychology 2002, 27:1-25.
29. Koenig K.A., Frey M.C., Detterman D.K. ACT and general cognitive ability. Intelligence 2008, 36:153-160.
30. Kulik J.A., Bangert-Drowns R.L., Kulik C.-L.C. Effectiveness of coaching for aptitude tests. Psychological Bulletin 1984, 95:179-188.
31. Kulik J.A., Kulik C.-L.C., Bangert R.L. Effects of practice on aptitude and achievement test scores. American Educational Research Journal 1984, 21:435-447.
32. Kyllonen P.C., Christal R.E. Reasoning ability is (little more than) working memory capacity?!. Intelligence 1990, 14:389-433.
33. Lievens F., Buyse T., Sackett P.R. Retest effects in operational selection settings: Developments and test of framework. Personnel Psychology 2005, 58:981-1007.
34. Lievens F., Peeters H., Schollaert E. Situational judgment tests: A review of recent research. Personnel Review 2007, 37:426-441.
35. Lievens F., Reeve C.L., Heggestad E.D. An examination of psychometric bias due to retesting on cognitive ability rests in selection settings. Journal of Applied Psychology 2007, 92:1672-1682.
36. Lipsey M.W., Wilson D.B. The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist 1993, 48:1181-1209.
37. Lynn R., Allik J., Irwing P. Sex differences on three factors identified in Raven's Standard Progressive Matrices. Intelligence 2004, 32:411-424.
38. Matton N., Vautier S., Raufaste E. Situational effects may account for gain scores in cognitive ability testing: A longitudinal SEM approach. Intelligence 2009, 37:412-421.
39. Meade A.W., Lautenschlager G.J., Hecht J.E. Establishing measurement equivalence and invariance in longitudinal data with item response theory. International Journal of Testing 2005, 5:279-300.
40. Meulders M., Xie Y. Person-by-item predictors. Explanatory item response models 2004, 214-240. Springer, New York. P. de Boeck, M. Wilson (Eds.).
41. Muthén L.K., Muthén B.O. Mplus [computer software] 2010, Muthén & Muthén, Los Angeles, CA.
42. Nevo B. The effects of general practice, specific practice, and item familiarization on change in aptitude test scores. Measurement and Evaluation in Guidance 1976, 9:16-20.
43. Nkaya H.N., Huteau M., Bonnet J.-P. Retest effect on cognitive performance on the Raven-38 matrices in France and in the Congo. Perceptual and Motor Skills 1994, 78:503-510.
44. Rabe-Hesketh S., Skrondal A. Multilevel and longitudinal modeling using Stata 2005, Stata Press, College Station, TX.
45. Rabe-Hesketh S., Skrondal A., Pickles A. GLLAMM manual. Technical Report 160 2004, Division of Biostatistics, University of California Berkeley. http://bepress.com/icbbiostat/paper160/.
46. Reeve C.L., Heggestad E.D., Lievens F. Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence 2009, 37:34-41.
47. Reeve C.L., Lam H. The psychometric paradox of practice effects due to retesting: Measurement invariance and stable ability estimates in the face of observed score changes. Intelligence 2005, 33:535-549.
48. Reeve C.L., Lam H. The relation between practice effects, test-taker characteristics and degree of g-saturation. International Journal of Testing 2007, 7:225-242.
49. Reise S.P., Smith L., Furr M. Invariance on the NEO-PI neuroticism scale. Multivariate Behavioral Research 2001, 36:83-110.
50. Rheinberg F., Vollmeyer R., Burns B.D. FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern-und Leistungssituationen [QCM: A questionnaire for the assessment of current motivation in learning situations]. Diagnostica 2001, 47:57-66.
51. Rost J., von Davier M. A conditional item fit index for Rasch models. Applied Psychological Measurement 1994, 18:171-182.
52. Sackett P.R., Burris L.R., Ryan A.M. Coaching and practice effects in personnel selection. International review of industrial and organizational psychology 1989, 145-183. Wiley, Chichester, NY. C.L. Cooper, I.T. Robertson (Eds.).
53. Skrondal A., Rabe-Hesketh S. Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models 2004, Chapman & Hall/CRC, Boca Raton, FL.
54. Skrondal A., Rabe-Hesketh S. Prediction in multilevel generalized linear models. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009, 172:659-687.
55. Sternberg R.J., Grigorenko E.L. Dynamic testing. The nature and measurement of learning potential 2002, Cambridge University Press, New York.
56. Taris T.W., Bok I.A., Meijer Z.Y. Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach. Journal of Psychology 1998, 132:301-316.
57. te Nijenhuis J., van Vianen A., van der Flier H. Score gains on g-loaded tests: No g. Intelligence 2007, 35:283-300.
58. te Nijenhuis J., Voskuijl O., Schijve N. Practice and coaching on IQ tests: Quite a lot of g. International Journal of Selection and Assessment 2001, 9:302-308.
59. Vandenberg R.J., Lance C.E. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2000, 3:4-70.
60. Vollmeyer R., Rheinberg F. Motivational effects on self-regulated learning with different tasks. Educational Psychology Review 2006, 18:239-253.
61. von Davier M. Winmira 2001 2001, Assessment Systems Corporation, St. Paul, MN.
62. Weiß R.H. Grundintelligenztest Skala 2 (CFT-20) mit Wortschatztest (WS) und Zahlenfolgentest (ZF) [Reasoning test scale 2 (CFT-20) with vocabulary scale and number series] 1998, Hogrefe, Göttingen, Ger.