A Proposed Framework for Conducting Data-Based Test Analysis

Psychological Methods

Volumn 13, Issue 4, 2008, Pages 376-390

  Slaney, Kathleen L ^a   Maraun, Michael D ^a  

^a SIMON FRASER UNIVERSITY   (Canada)

Author keywords

assessment of psychometric properties of tests; data based test analytic framework; external test validity; internal test validity; test analysis

Indexed keywords

ARTICLE; HUMAN; PSYCHOLOGIC TEST; PSYCHOLOGY; PSYCHOMETRY; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS;

DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOLOGICAL TESTS; PSYCHOLOGY, EXPERIMENTAL; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 57749120481     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0014269     Document Type: Article

References (81)

1. American Educational Research Association [AERA], American Psychological Association, and National Council on Measurement in Education . Standards for educational and psychological testing (1999), American Educational Research Association, Washington, DC
2. American Psychological Association [APA] . Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin 51 Pt. 2 (1954) 1-38
3. Barrett P. Structural equation modeling: Adjudging model fit. Personality and Individual Differences 25 (2007) 815-824
4. Beck A.T., Steer R.A., and Brown G.K. Manual for Beck Depression Inventory-II (1996), Psychological Corporation, San Antonio, TX
5. Birnbaum A. Some latent trait models and their use in inferring an examinee's ability. In: Lord F., and Novick M. (Eds). Statistical theories of mental test scores (1968), Addison-Wesley, Reading, MA 397-479
6. Blalock H.M. Causal inferences in nonexperimental research (1964), University of North Carolina Press, Chapel Hill
7. Blinkhorn S.F. Past imperfect, future conditional: Fifty years of test theory. British Journal of Mathematical and Statistical Psychology 50 (1997) 175-186
8. Bollen K.A., and Lennox R. Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin 110 (1991) 305-314
9. Borsboom D., Mellenbergh G.J., and van Heerden J. The theoretical status of latent variables. Psychological Review 110 (2003) 203-219
10. Borsboom D., Mellenbergh G.J., and van Heerden J. The concept of validity. Psychological Review 111 (2004) 1061-1071
11. Campbell D.T., and Fiske D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56 (1959) 81-105
12. Campbell D.T., and Stanley J.C. Experimental and quasi-experimental designs for research (1963), Rand McNally, Chicago
13. Clogg C.C. Latent class models. In: Arminger G., Clogg C.C., and Sobel M.E. (Eds). Handbook of statistical modeling for the social and behavioral sciences (1995), Plenum, New York 311-360
14. Costa Jr. P.T., and McRae R.R. The NEO Personality Inventory manual (1985), Psychological Assessment Resources, Odessa, FL
15. Cronbach L.J. Test validation. In: Thorndike R.L. (Ed). Educational measurement. 2nd ed. (1971), American Council on Education, Washington, DC 443-507
16. Cronbach L.J. Five perspectives on the validity argument. In: Wainer H., and Braun H. (Eds). Test validity (1988), Erlbaum, Hillsdale, NJ 3-17
17. Cronbach L.J. Construct validation after thirty years. In: Linn R.L. (Ed). Intelligence: Measurement theory and public policy. Proceedings of a symposium in honor of Lloyd G. Humphreys (1989), University of Illinois Press, Urbana 147-171
18. Cronbach L.J., and Meehl P.E. Construct validity and psychological tests. Psychological Bulletin 52 (1955) 281-302
19. Edwards J.R., and Bagozzi R.P. On the nature and direction of relationships between constructs and measures. Psychological Methods 5 (2000) 155-174
20. Embretson S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 93 (1983) 179-197
21. Fornell C., and Bookstein F.L. Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research 19 (1982) 440-452
22. Grayson D. Two-group classification in latent trait theory: Scores with monotone likelihood ratio. Psychometrika 53 (1988) 383-392
23. Green S.B., Lissitz R.W., and Mulaik S.A. Limitations of coefficient alpha as an index of unidimensionality. Educational and Psychological Measurement 37 (1977) 827-838
24. Guttman L. The determinacy of factor score matrices with implications for five other basic problems of common-factor theory. British Journal of Statistical Psychology 8 (1955) 17-24
25. Hamaker E.L., Dolan C.V., and Molenaar P.C.M. Statistical modeling of the individual: Rationale and application of multivariate time series analyses. Multivariate Behavioral Research 40 (2005) 207-233
26. Hattie J.A. An empirical study of various indices for determining undimensionality. Multivariate Behavioral Research 19 (1984) 49-78
27. Hattie J.A. Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement 9 (1985) 139-164
28. Heinen T. Latent class and discrete latent trait models: Similarities and differences (1996), Sage, Thousand Oaks, CA
29. Hogan T.P., and Agnello J. An empirical study of reporting practices concerning measurement validity. Educational and Psychological Measurement 64 4 (2004) 802-812
30. Holland P.W. On the sampling theory foundations of item response theory models. Psychometrika 55 (1990) 577-601
31. Holland P.W., and Rosenbaum P. Conditional association and unidimensionality in monotone latent variable models. Annals of Statistics 14 4 (1986) 1523-1543
32. Jöreskog K. Testing a simple structure hypothesis in factor analysis. Psychometrika 3 (1966) 165-178
33. Jöreskog K. A general approach to confirmatory maximum likelihood factor analysis. Psychometrika 34 (1969) 183-202
34. Kane M.T. An argument-based approach to validity. Psychological Bulletin 112 (1992) 527-535
35. Kane M.T. Validation. In: Brennan R.L. (Ed). Educational measurement. 4th ed. (2006), American Council on Education/Praeger, Washington, DC 17-64
36. Lazarsfeld P.F. The logical and mathematical foundation of latent structure analysis. In: Stouffer S.A., et al. (Ed). Measurement and prediction (1950), Princeton University Press, Princeton 362-412
37. Lazarsfeld P.F., and Henry N.W. Latent structure analysis (1968), Houghton Mifflin, New York
38. Lee S., and Hershberger S. A simple rule for generating equivalent models in covariance structure modeling. Multivariate Behavioral Research 25 (1990) 313-334
39. Loevinger J. Objective tests as instruments of psychological theory. Psychological Reports 3 (1957) 635-694
40. Lord F.M. A theory of test scores. Psychometric Monographs No. 7 (1952), Psychometric Society, Iowa City, IA
41. Lord F.M. The relation of test score to the trait underlying the test. Educational and Psychological Measurement 13 (1953) 517-549
42. Maraun M.D., Jackson J.S.H., Luccock C.R., Belfer S.E., and Chrisjohn R.D. CA and SPOD for the analysis of test comprised of binary items. Educational and Psychological Measurement 58 (1998) 916-928
43. Maraun M.D., Slaney K., and Goddyn L. An analysis of Meehl's MAXCOV-HITMAX procedure for the case of dichotomous indicators. Multivariate Behavioral Research 38 (2003) 81-112
44. McDonald R.P. Factor interaction in non-linear factor analysis. British Journal of Mathematical and Statistical Psychology 20 (1967) 205-215
45. McDonald R.P. Test theory: A unified treatment (1999), Erlbaum, Mahwah, NJ
46. McDonald R.P., and Burr E.J. A comparison of four methods of constructing factor scores. Psychometrika 34 (1967) 381-401
47. Meier S.T., and Davis S.R. Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology 37 (1990) 113-115
48. Mellenbergh G.J. Generalized linear item response theory. Psychological Bulletin 115 (1994) 300-307
49. Mellenbergh G.J. Measurement precision in test score and item response models. Psychological Methods 1 (1996) 293-299
50. Messick S. Test validity and the ethics of assessment. American Psychologist 35 (1980) 1021-1027
51. Messick S. The once and future issues of validity: Assessing the meaning and consequences of measurement. In: Wainer H., and Braun H.I. (Eds). Test validity (1988), Erlbaum, Hillsdale, NJ 33-46
52. Messick S. Validity. In: Linn R.L. (Ed). Educational measurement. 3rd ed. (1989), MacMillan, New York 13-103
53. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific enquiry into score meaning. American Psychologist 50 (1995) 741-749
54. Messick S. Test validity: A matter of consequence. Social Indicators Research 45 (1998) 35-44
55. Mislevy R.J., Steinberg L.S., and Almond R.G. On the structure of educational assessment. Measurement: Interdisciplinary Research and Perspectives 1 (2003) 3-62
56. Molenaar P.C.M. A dynamic factor model for the analysis of multivariate tiem series. Psychometrika 50 (1985) 181-202
57. Molenaar P.C.M. Dynamic latent variable models in developmental psychology. In: von Eye A., and Clogg C.C. (Eds). Latent variables analysis: Applications for developmental research (1994), Sage, Thousand Oaks, CA 155-180
58. Molenaar P.C.M., Huizenga H.M., and Nesselroade J.R. The relationship between the structure of interindividual and intraindividual variability: A theoretical and empirical vindication of developmental systems theory. In: Staudinger U.M., and Lindenberger U. (Eds). Understanding human development: Dialogues with lifespan psychology (2003), Kluwer Academic, Dordrecht, the Netherlands 339-360
59. Muraki E. Fitting a polytomous item response model to Likert-type data. Applied Psychological Measurement 14 (1990) 59-71
60. Peak H. Problems of objective observation. In: Festinger L., and Katz D. (Eds). Research methods in the behavioral sciences (1953), Holt, Rinehart & Winston, New York 243-299
61. Rasch G. Probabilistic models for some intelligence and attainment tests (1960), Danish Institute for Educational Research, Copenhagen
62. Raykov T., and Penev S. On structural equation model equivalence. Multivariate Behavioral Research 34 (1999) 199-244
63. Roskam E.E., and Ellis J. "The irrelevance of factor analysis for the study of group differences": Commentary. Multivariate Behavioral Research 27 (1992) 205-218
64. Rovine M.J., and Molenaar P.C.M. Relating factor models for longitudinal data to quasi-simplex and NARMA models. Multivariate Behavioral Research 40 (2005) 83-114
65. Samejima F. Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph 17 Suppl. 4 (1969)
66. Samejima F. The graded response model. In: van der Linden W.J., and Hambleton R.K. (Eds). Handbook of modern item response theory (1996), Springer, New York 85-100
67. Schonemann P.H., and Wang M.-M. Some new results on factor indeterminacy. Psychometrika 37 Pt. 1 (1972) 61-91
68. Shepard L.A. Evaluating test validity. Review of Research in Education 19 (1993) 405-450
69. Slaney, K. L. (2006). The logic of test analysis: An evaluation of test theory and a proposed logic for test analysis. Unpublished doctoral dissertation, Simon Fraser University, Burnaby, British Columbia, Canada.
70. Spearman C. "General intelligence," objectively determined and measured. American Journal of Psychology 15 (1904) 201-293
71. Steer R.A., Ranieri W.F., Kumar G., and Beck A.T. Beck Depression Inventory-II items associated with self-reported symptoms of ADHD in adult psychiatric outpatients. Journal of Personality Assessment 80 (2003) 58-63
72. Steiger J.H., and Schonemann P.H. A history of factor indeterminacy. In: Shye S. (Ed). Theory construction and data analysis (1978), Jossey-Bass, San Franciso 136-178
73. Stelzl I. Changing a causal hypothesis without changing the fit: Some rules for generating equivalent path models. Multivariate Behavioral Research 21 (1986) 309-331
74. Tellegen, A. (1982). Brief manual for the Multidimensional Personality Questionnaire. Unpublished manuscript, University of Minnesota.
75. Thissen D., and Steinberg L. A taxonomy of item response models. Psychometrika 51 (1986) 567-577
76. Thissen D., Steinberg L., Pyszczynski T., and Greenberg J. An item response theory for personality and attitude scales: Item analysis using restricted factor analysis. Applied Psychological Measurement 7 (1983) 211-226
77. Vacha-Haase T., Ness C., Nilsson J., and Reetz D. Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education 67 (1999) 335-341
78. van Rijn P.W., and Molenaar P.C.M. Logistic models for single subject time series analysis. In: van der Ark L.A., Croon M.A., and Sijtsma K. (Eds). New developments in categorical data analysis for the social and behavioral sciences (2005), Erlbaum, Mahwah, NJ 125-145
79. van Schuur W.H., and Kiers H.A. Why factor analysis often is the incorrect model of analyzing bipolar concepts, and what model to use instead. Applied Psychological Measurement 18 (1994) 97-110
80. Wainer H. Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin 83 (1976) 213-217
81. Whittington D. How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement 58 (1998) 21-37