The effects of the number of scale points and non-normality on the generalizability coefficient: A Monte Carlo study

Applied Measurement in Education

Volumn 20, Issue 4, 2007, Pages 357-376

  Shumate, Steven R ^a   Surles, James ^b   Johnson, Robert L ^c   Penny, Jim ^d  

^a AUBURN UNIVERSITY   (United States)

^b TEXAS TECH UNIVERSITY   (United States)

^c UNIVERSITY OF SOUTH CAROLINA   (United States)

^d CASTLE Worldwide Inc

Author keywords

[No Author keywords available]

Indexed keywords

EID: 41449118886     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957340701429645     Document Type: Article

References (39)

1. Barnette, J. J. (1999). Nonattending respondent effects on internal consistency of self-administered surveys: A Monte Carlo simulation study. Educational and Psychological Measurement, 59(1), 38-46.
2. Bond, L. (1998). Culturally responsive pedagogy and the assessment of accomplished teaching. Journal of Negro Education, 67(3), 242-254.
3. Brennan, R. (2001). Generalizability theory. New York: Springer.
4. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277-289.
5. Charles, R., Lester, F., & O'Daffer, P. (1987). How to evaluate progress in problem solving. Reston, VA: National Council of Teachers of Mathematics.
6. Coffinan, W. (1971). On the reliability of ratings of essay examinations in English. Research in the Teaching of English, 5, 24-36.
7. Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York: Academic Press.
8. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability scores and profiles. New York: Wiley.
9. Davison, M. L., & Sharina, A. R. (1988). Parametric statistics and levels of measurement. Psychological Bulletin, 104, 137-144.
10. Davison, M. L., & Sharma, A. R. (1990). Parametric statistics and levels of measurement: Factorial designs and multiple regression. Psychological Bulletin, 107, 394-400.
11. Educational Testing Service. (1999). Technical analysis report. Princeton, NJ: Author.
12. Godshalk, F., Swineford, F., & Coffman, W. (1966). The measurement of writing ability. Princeton, NJ: College Entrance Examination Board.
13. Gosling, G. (1966). Marking English compositions. Victoria: Australian Council for Educational Research.
14. Hsu, T. C., & Feldt, L. S. (1969). The effect of limitations on the number of criterion score values on the significance level of the F-test. American Education Research Journal, 6(4), 515-527.
15. Huitema, B. E., McKean, J. W., & McNight, S. (1999). Autocorrelation effects on least-squares intervention analysis of short time series. Educational and Psychological, Measurement, 59(5), 767-786.
16. Kenny, P. A., & Silver, E. A. (1993). Student self-assessment in mathematics. In N. L. Webb & A. F. Coxford (Eds.), Assessment in the mathematics classroom, 1193 yearbook of the national council of teachers in mathematics (NCTM) (pp. 229-238). Reston, VA: NCTM.
17. Kirk, R. E. (1990). Statistics: An introduction (3rd ed.). Fort Worth, TX: Holt, Rinehart and Winston.
18. Lindquist, E. F. (1953). Design and analysis experiments in psychology. Boston: Houghton Mifflin.
19. Longford, N. T. (1994). Reliability of essay rating and score adjustment. Journal of Education and Behavioral Statistics, 19(3), 171-200.
20. Luce, R. D., Krantz, D. H., Suppes, P., & Tversky, A. (1990). Foundations of measurement (Vol. 3). New York: Academic Press.
21. Marzano, R. J., Pickering, D., & McTighe, J. (1993). Assessing student outcomes: Performance assessment using the dimensions of learning model. Alexandria, VA: Association for Supervision and Curriculum Development.
22. Maxwell, S. E., & Delaney, H. D. (1985). Measurement scales and statistics: An examination of variable validity. Psychological Bulletin, 97, 85-93.
23. McColly, W., & Remstad, R. (1965). Composition rating scales for general merit: An experimental evaluation. Journal of Educational Research 59, 55-56.
24. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105(1), 156-166.
25. Mitchell, J. (1986). Measurement scales and statistics: A clash of paradigms. Psychological Bulletin, 100(3), 398-407.
26. National Board for Professional Teaching Standards (2001). Early and middle childhood art: Scoring guide. Washington, DC: Author.
27. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
28. Penny, J., & Surles, J. (2002). SAS code for simulating the reliability for two raters and breaking those ratings into different scale points. Jamestown, NC: Author.
29. Penny, J., Johnson, R. L., & Gordon, B. (2000a). The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric. Assessing Writing, 7, 143-164.
30. Penny, J., Johnson, R. L., & Gordon, B. (2000b). Using rating augmentation to expand the scale of an analytic rubric. The Journal of Experimental Education., 68(3), 269-287.
31. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: Sage Publications.
32. Smith, P. (1978). Sampling errors of variance components in small sample multifacet generalizability studies. Journal of Educational Statistics, 3, 319-346.
33. Srivastava, A. B. L. (1959). Effect of non-normality on the power of the analysis of variance test. Biometrika, 46, 114-122.
34. Stenmark, J. K. (1989). Assessment alternatives in mathematics: An overview of assessment techniques that promote learning. Berkeley: EQUALS, University of California.
35. Tiku, M. L. (1964). Approximating the general non-normal variance-ratio sampling distributions. Biometrika, 51, 83-95.
36. Tiku, M. L. (1971). Power function of the F-test under non-normal situations. Journal of the American Statistical Association, 66, 913-916.
37. Tiku, M. L. (1975). Laguerre series forms of the distributions of classical test statistics and their robustness in non-normal situations. In R. P. Gupta (Ed.), Applied statistics (pp. 333-350). Amsterdam: North-Holland.
38. Townsend, J. T., & Ashby, F. G. (1984). Measurement scales and statistics: The misconception misconceived. Psychological Bulletin, 96(2), 394-401.
39. Van der Kamp, L. J. Th. (1976). Generalizability and educational measurement. In D. N. M. De Gruijter & L. J. Th. Van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 173-184). New York: Wiley.