The assumption of a reliable instrument and other pitfalls to avoid when considering the reliability of data

Frontiers in Psychology

Volumn 3, Issue APR, 2012, Pages

  Nimon, Kim ^a   Zientek, Linda Reichwein ^b   Henson, Robin K ^a  

^a UNIVERSITY OF NORTH TEXAS   (United States)

^b SAM HOUSTON STATE UNIVERSITY   (United States)

Author keywords

Correlated error; Measurement error; Reliability

Indexed keywords

EID: 84867044884     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2012.00102     Document Type: Review

References (69)

1. Allen, M. J., and Yen, W. M. (1979). Introduction to Measurement Theory. Monterey, CA: Brooks/Cole.
2. American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educ. Res. 35, 33-40.
3. American Psychological Association. (2009a). Publication Manual of the American Psychological Associa-tion: Supplemental Material: Writ-ing Clearly and Concisely, Chap. 3. Available at: http://supp.apa.org/style/pubman-ch03.00.pdf.
4. American Psychological Association. (2009b). Publication Manual of the American Psychological Association, 6th Edn. Washington, DC: American Psychological Association.
5. Anastasi, A., and Urbina, S. (1997). Psychological Testing, 7th Edn. Upper Saddle River, NJ: Prentice Hall.
6. Bandura, A. (2006). "Guide for constructing self-efficacy scales," in Self-Efficacy Beliefs of Adolescents, Vol. 5, eds F. Pajares and T. Urdan (Greenwich, CT: Information Age Publishing), 307-337.
7. Barrett, P. (2001). Assessing the Reliability of Rating Data. Available at: http://www.pbarrett.net/presentations/rater.pdf.
8. Beck, A. T. (1990). Beck Anxiety Inven-tory (BAI). San Antonio, TX: The Psychological Corporation.
9. Beck, A. T. (1996). Beck Depression Inventory-II (BDI-II). San Antonio, TX: The Psychological Corporation. Benefield, D. (2011). SAT Scores by State, 2011. Harrisburg, PA: Common Wealth Foundation. Available at: http://www.commonwealthfoundation.org/policyblog/detail/sat-scores-by-st ate-2011.
10. Charles, E. P. (2005). The correction for attenuation due to measurement error: clarifying concepts, and creating confidence sets. Psychol. Methods 10, 206-226.
11. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educ. Psy-chol. Meas. 20, 37-46.
12. College Board. (2011). Archived SAT Data and Reports. Available at: http://professionals.collegeboard.com/data-reports-research/sat/archived .
13. Crocker, L., and Algina, J. (1986). Introduction to Classical and Mod-ern Test Theory. Belmont, CA: Wadsworth.
14. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika 16, 297-334.
15. Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educ. Psychol. Meas. 64, 391-418. (editorial assistance by Shavelson, R. J.).
16. Dillon, W. R., and Bearden, W. (2001). Writing the survey question to capture the concept. J. Consum. Psychol. 10, 67-69.
17. Dimitrov, D. M. (2002). Reliability: arguments for multiple perspectives and potential problems with generalizability across studies. Educ. Psy-chol. Meas. 62, 783-801.
18. Dozois, D. J. A., Dobson, K. S., and Ahnberg, J. L. (1998). A psychometric evaluation of the Beck Depression Inventory -II. Psychol. Assess. 10, 83-89.
19. Faul, F., Erdfelder, E., Lang, A., and Buchner, A. (2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175-191.
20. Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions, 2nd Edn. New York: John Wiley.
21. Harman, H. H. (1967). Modern Fac-tor Analysis. Chicago: University of Chicago Press.
22. Henson, R. K. (2001). Understanding internal consistency reliability estimates: a conceptual primer on coefficient alpha. Meas. Eval. Couns. Dev. 34, 177-189.
23. Hogan, T. P., Benjamin, A., and Brezinski, K. L. (2000). Reliability methods: a note on the frequency of use of various types. Educ. Psychol. Meas. 60, 523-531.
24. Hulin, C., Netemeyer, R., and Cudeck, R. (2001). Can a reliability coefficient be too high? J. Consum. Psychol. 10, 55-58.
25. Hunter, J. E., and Schmidt, F. L. (1990). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Newbury Park, CA: Sage.
26. Kieffer, K. M., Reese, R. J., and Thompson, B. (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: a methodological review. J. Exp. Educ. 69, 280-309.
27. Leach, L. F., Henson, R. K., Odom, L. R., and Cagle, L. S. (2006). A reliability generalization study of the Self-Description Questionnaire. Educ. Psychol. Meas. 66, 285-304.
28. Lorenzo-Seva, U., Ferrando, P. J., and Chico, E. (2010). Two SPSS programs for interpreting multiple regression results. Behav. Res. Meth-ods 42, 29-35.
29. Magnusson, D. (1967). Test Theory. Reading, MA: Addison-Wesley.
30. Marat, D. (2005). Assessing mathematics self-efficacy of diverse students form secondary schools in Auckland: implications for academic achievement. Issues Educ. Res. 15, 37-68.
31. Marsh, H. W. (1989). Age and sex effects in multiple dimensions of self-concept: preadolescence to early adulthood. J. Educ. Psychol. 81, 417-430.
32. Miller, C. S., Shields, A. L., Campfield, D., Wallace, K. A., and Weiss, R. D. (2007). Substance use scales of the Minnesota Multiphasic Personality Inventory: an exploration of score reliability via metaanalysis. Educ. Psychol. Meas. 67, 1052-1065.
33. Muchinsky, P. M. (1996). The correction for attenuation. Educ. Psychol. Meas. 56, 63-75.
34. Nimon, K., and Henson, R. K. (2010). Validity of residualized variables after pretest covariance corrections: still the same variable? Paper Pre-sented at the Annual Meeting of the American Educational Research Asso-ciation, Denver, CO.
35. Nunnally, J. C. (1967). Psychometric Theory. New York: McGraw-Hill.
36. Nunnally, J. C. (1978). Psychomet-ric Theory, 2nd Edn. New York: McGraw-Hill.
37. Nunnally, J. C., and Bernstein, I. H.(1994). Psychometric Theory, 3rd Edn. New York: McGraw-Hill.
38. Onwuegbuzie, A. J., Roberts, J. K., and Daniel, L. G. (2004). A proposed new "what if" reliability analysis for assessing the statistical significance of bivariate relationships. Meas. Eval. Couns. Dev. 37, 228-239.
39. Osborne, J., and Waters, E. (2002). Four assumptions of multiple regression that researchers should always test. Pract. Assess. Res. Eval. 8. Available at: http://PAREonline.net/getvn.asp?v=8&n=2.
40. Pajares, F., and Graham, L. (1999). Self-efficacy, motivation constructs, and mathematics performance of entering middle school students. Contemp. Educ. Psychol. 24, 124-139.
41. Pedhazur, E. J. (1997). Multiple Regres-sion in Behavioral Research: Expla-nation and Prediction, 3rd Edn. Fort Worth, TX: Harcourt Brace.
42. Pintrich, P. R., Smith, D. A. F., Garcia, T., and McKeachie, W. J. (1991). A Manual for the Use of the Moti-vated Strategies for Learning Ques-tionnaire (MSLQ). Ann Arbor: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.
43. Popham, W. J. (2000). Modern Educa-tional Measurement: Practical Guidelines for Educational Leaders, 3rd Edn. Needham, MA: Allyn & Bacon.
44. Public Agenda. (2011). State-by-State SAT and ACT Scores. Available at: http://www.publicagenda.org/charts/state-state-sat-and-act-scores.
45. Rodriguez, M. C., and Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychol. Methods, 11, 306-322.
46. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638-641.
47. Rosenthal, R. (1995). Writing metaanalytic reviews. Psychol. Bull. 118, 183-192.
48. Schmidt, F. L., and Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. J. Appl. Psychol. 62, 529-540.
49. Shavelson, R., and Webb, N. (1991). Generalizability Theory: A Primer. Newbury Park, CA: Sage.
50. Shields, A. L., and Caruso, J. C. (2004). A reliability induction and reliability generalization study of the Cage Questionnaire. Educ. Psychol. Meas. 64, 254-270.
51. Spearman, C. (1904). The proof and measurement of association between two things. Am.J.Psychol. 15, 72-101.
52. Stemler, S. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Pract. Assess. Res. Eval. 9. Available at: http://PAREonline.net/getvn.asp?v=9&n=4.
53. Thompson, B. (2000). "Ten commandments of structural equation modeling," in Reading and Understanding More Multivari-ate Statistics, eds L. Grimm and P. Yarnold (Washington, DC: American Psychological Association), 261-284.
54. Thompson, B. (2003a). "Guidelines for authors reporting score reliability estimates," in Score Relia-bility: Contemporary Thinking on Reliability Issues, ed. B. Thompson (Newbury Park, CA: Sage), 91-101.
55. Thompson, B. (2003b). "A brief introduction to generalizability theory," in Score Reliability: Contemporary Thinking on Reliability Issues, ed. B. Thompson (Newbury Park, CA: Sage), 43-58.
56. Thompson, B., and Vacha-Haase, T. (2000). Psychometrics is datametrics: the test is not reliable. Educ. Psychol. Meas. 60, 174-195.
57. Trafimow, D., and Rice, S. (2009). Potential performance theory (PPT): describing a methodology for analyzing task performance. Behav. Res. Methods 41, 359-371.
58. Vacha-Haase, T. (1998). Reliability generalization: exploring variance in measurement error affecting score reliability across studies. Educ. Psy-chol. Meas. 58, 6-20.
59. Vacha-Haase, T., Henson, R. K., and Caruso, J. C. (2002). Reliability generalization: moving toward improved understanding and use of score reliability. Educ. Psychol. Meas. 62, 562-569.
60. Vacha-Haase, T., Kogan, L, R., and Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: validity of score reliability inductions. Educ. Psychol. Meas. 60, 509-522.
61. Vacha-Haase, T., Ness, C., Nillson, J., and Reetz, D. (1999). Practices regarding reporting of reliability coefficients: a review of three journals. J. Exp. Educ. 67, 335-341.
62. Vacha-Haase, T., and Thompson, B. (2011). Score reliability: a retrospective look back at 12 years of reliability generalization studies. Meas. Eval. Couns. Dev. 44, 159-168.
63. Wetcher-Hendricks, D. (2006). Adjustments to the correction for attenuation. Psychol. Methods 11, 207-215.
64. Wilkinson, L., and APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: guidelines and explanation. Am. Psy-chol. 54, 594-604.
65. Yetkiner, Z. E., and Thompson, B. (2010). Demonstration of how score reliability is integrated into SEM and how reliability affects all statistical analyses. Mult. Linear Regress.Viewp. 26, 1-12.
66. Zientek, L. R., Capraro, M. M., and Capraro, R. M. (2008). Reporting practices in quantitative teacher education research: one look at the evidence cited in the AERA panel report. Educ. Res. 37, 208-216.
67. Zientek, L. R., and Thompson, B. (2009). Matrix summaries improve research reports: secondary analyses using published literature. Educ. Res. 38, 343-352.
68. Zimmerman, D. W. (2007). Correction for attenuation with biased reliability estimates and correlated errors in populations and samples. Educ. Psychol. Meas. 67, 920-939.
69. Zimmerman, D. W., and Williams, R. H. (1977). The theory of test validity and correlated errors of measurement. J. Math. Psychol. 16, 135-152.