Measurement choices: Reliability, validity, and generalizability

Handbook of Ethics in Quantitative Methodology

Volumn , Issue , 2011, Pages 127-157

  Carrig, Madeline M ^a   Hoyle, Rick H ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84873155499     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (71)

1. Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D.,... Lang, T. (2001). The Revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine, 134, 663-694.
2. American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35, 33-40.
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. American Psychological Association. (1954). Technical recommendations for psychological tests and diagnostic techniques. Washington, DC: Author.
5. American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060-1073.
6. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
7. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology:Why do we need them? What might they be? American Psychologist, 63, 839-851.
8. Arkes, H. R., & Tetlock, P. E. (2004). Attributions of implicit prejudice, or "Would Jesse Jackson 'fail' the Implicit Association Test?" Psychological Inquiry, 15, 257-278.
9. Bailar, J. C., III, & Mosteller, F. (1988). Guidelines for statistical reporting in articles for medical journals: Amplifications and explanations. Annals of Internal Medicine, 108, 266-273.
10. Barbour, V., Moher, D., Sox, H., & Kahn, M. (2005). Standards of reporting biomedical research: What's new? Science Editor, 28, 4.
11. Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370-379.
12. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
13. Blackman, N. J.-M., & Koval, J. J. (1993). Estimating rater agreement in 2 x 2 tables: Corrections for chance and intraclass correlation. Applied Psychological Measurement, 17, 211-223.
14. Blanton, H., Jaccard, J., Gonzales, P. M., & Christie, C. (2006). Decoding the Implicit Association Test: Implications for criterion prediction. Journal of Experimental Social Psychology, 42, 192-212.
15. Bosson, J. K., Swann, W. B., & Pennebaker, J. W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631-643.
16. Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M.,... de Vet, H. C. (2003). Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Annals of Internal Medicine, 138, 40-44.
17. Brennan, R. L. (2001). Generalizability theory. New York: Springer-Verlag.
18. Brennan, R. L. (2006). Perspectives on the evolution and future of educational measurement. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 1-16). Westport, CT: Praeger Publishers.
19. Cai, L., Maydeu-Olivares, A., Coffman, D. L., & Thissen, D. (2006). Limitedinformation goodness-of-fit testing of item response theory models for sparse 2p tables. British Journal of Mathematical and Statistical Psychology, 59, 173-194.
20. Chedd, G. (Writer and Director). (2007). The hidden prejudice [Television series episode]. In J. Angier & G. Chedd (Executive Producers), Scientific American Frontiers. Public Broadcasting Corporation.
21. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
22. Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Miffl in.
23. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
24. Des Jarlais, D. C., Lyles, C., & Crepaz, N. (2004). Improving the reporting quality of nonrandomized evaluations of behavior and public health interventions:The TREND statement. American Journal of Public Health, 94, 361-366.
25. DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327-346.
26. Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5, and 7 response categories:A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47, 309-326.
27. Dunivant, N. (1981). The effects of measurement error on statistical models for analyzing change: Final report (Grant NIE-G-78-0071). Washington, DC: National Institute of Education (Educational Resources Information Center Document Reproduction Service No. ED223680). Retrieved from Educational Resources Information Center database.
28. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York: American Council on Education and Macmillan.
29. Feldt, L. S., & Charter, R. A. (2003). Estimating the reliability of a test split into two parts of equal or unequal length. Psychological Methods, 8, 102-109.
30. Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, selfesteem, and stereotypes. Psychological Review, 102, 4-27.
31. Greenwald, A. G., McGhee, D. E., & Schwartz, J. K. L. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.
32. Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197-216.
33. Greenwald, A. G., Poehlman, T. A., Uhlmann, E., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17-41.
34. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: Praeger Publishers.
35. Hägglund, G. (1982). Factor analysis by instrumental variables. Psychometrika, 47, 209-222.
36. Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12, 38-47.
37. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL:Academic Press.
38. Hoyle, R. H., & Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 195-222). Thousand Oaks, CA: Sage Publications.
39. Huitema, B. E. (1980). The analysis of covariance and alternatives. New York: Wiley.
40. Jöreskög, K. G. (1970). A general method for analysis of covariance structures. Biometrika, 57, 239-251.
41. Kane, M. (1996). The precision of measurements. Applied Measurement in Education, 9, 355-379.
42. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: Praeger Publishers.
43. Koocher, G. P., & Keith-Spiegel, P. (2008). Ethics in psychology and the mental health professions: Standards and cases (3rd ed.). New York: Oxford University Press.
44. Lees-Haley, P. R. (1996). Alice in validityland, or the dangerous consequences of consequential validity. American Psychologist, 51, 981-983.
45. Lord, F. M. (1953). The relation of test score to the trait underlying the test. Educational and Psychological Measurement, 13, 517-548.
46. Lord, F. M. (1968). An analysis of the Verbal Scholastic Aptitude Test using Birnbaum's three-parameter logistic model. Educational and Psychological Measurement, 28, 989-1020.
47. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
48. Marcus-Roberts, H. M., & Roberts, F. S. (1987). Meaningless statistics. Journal of Educational Statistics, 12, 383-394.
49. Marshall, M., Lockwood, A., Bradley, C., Adams, C., Joy, C., & Fenton, M. (2000). Unpublished rating scales: A major source of bias in randomized controlled trials of treatments for schizophrenia. British Journal of Psychiatry, 176, 249-252.
50. Maxwell, S. E., & Delaney, H. D. (2004). Designing experiments and analyzing data:A model comparison perspective (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
51. Meade, A. W., & Lautenschlager, G. J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies in establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388.
52. Mellenbergh, G. J. (1989). Item bias and item response theory. International Journal of Educational Research, 13, 127-143.
53. Messick, S. (1988a). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.
54. Messick, S. (1988b). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. Braun (Eds.), Test validity (pp. 33-45). Hillsdale, NJ: Lawrence Erlbaum.
55. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education and Macmillan.
56. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50, 741-749.
57. Michell, J. (1986). Measurement scales and statistics: A clash of paradigms. Psychological Bulletin, 100, 398-407.
58. Olson, M. A., & Fazio, R. H. (2004). Reducing the infl uence of extra-personal associations on the Implicit Association Test: Personalizing the IAT. Journal of Personality and Social Psychology, 86, 653-667.
59. Project Implicit. (n.d.). Retrieved from http://projectimplicit.net/generalinfo.php
60. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.
61. Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356-442). Washington, DC: American Council on Education.
62. Thompson, B., & Daniel, L. G. (1996). Seminal readings on reliability and validity:A "hit parade" bibliography. Educational and Psychological Measurement, 56, 741-745.
63. Thompson, J. (2009). Project Implicit: Am I racist? Retrieved from http://www.myfoxchicago. com/dpp/news/project-implicit-am-i-a-racist-dpgo-20091029- jst1256860352012
64. Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement (pp. 560-620). Washington, DC: American Council on Education.
65. Tierney, J. (2008, November 17). In bias test, shades of gray. The New York Times. Retrieved from http://www.nytimes.com/2008/11/18/science/18tier.html
66. Townsend, J. T., & Ashby, F. G. (1984). Measurement scales and statistics: The misconception misconceived. Psychological Bulletin, 96, 394-401.
67. Vedantam, S. (2005, January 23). See no bias. The Washington Post, p. W12.
68. Wilkinson, L., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
69. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58-79.
70. Wright, T. A., & Wright, V. P. (2002). Organizational researcher values, ethical responsibility, and the committed-to-participant research perspective. Journal of Management Inquiry, 11, 173-185.
71. Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 111-153). Westport, CT: Praeger Publishers.