Generalizability theory: A practical guide to study design, implementation, and interpretation

Journal of School Psychology

Volumn 52, Issue 1, 2014, Pages 13-35

  Briesch, Amy M ^a   Swaminathan, Hariharan ^b   Welsh, Megan ^b   Chafouleas, Sandra M ^b  

^a NORTHEASTERN UNIVERSITY   (United States)

^b UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

Dependability; Generalizability theory; Measurement; School based assessment

Indexed keywords

HUMAN; METHODOLOGY; PSYCHOLOGY; PSYCHOMETRY; REPRODUCIBILITY; SCHOOL;

HUMANS; PSYCHOLOGY, EDUCATIONAL; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SCHOOLS;

EID: 84894901832     PISSN: 00224405     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jsp.2013.11.008     Document Type: Article

References (86)

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education Standards for educational and psychological testing 1999, Author, Washington, D.C.
2. Baraldi A.N., Enders C.K. An introduction to modern missing data analyses. Journal of School Psychology 2010, 48:5-37.
3. Bergeron R., Floyd R.G., McCormack A.C., Farmer W.L. The generalizability of externalizing behavior composites and subscale scores across time, rater, and instrument. School Psychology Review 2008, 37:91-108.
4. Bonnet D.G. Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine 2002, 21:1331-1335.
5. Box J.F., Fisher R.A. The life of a scientist 1978, Wiley, New York, NY.
6. Brennan R.L. An NCME instructional model on Generalizability Theory. Instructional topics in educational measurement, Module 14 1992, Winter, National Council on Measurement in Education, Madison, WI.
7. Brennan R.L. (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice 2000, 19:5-10.
8. Brennan R.L. Generalizability theory 2001, Springer-Verlag, New York, NY.
9. Brennan R.L. mGENOVA [Computer software and manual] 2001, University of Iowa, Iowa City, IA.
10. Brennan R.L. urGENOVA [Computer software and manual] 2001, University of Iowa, Iowa City, IA.
11. Briesch A.M., Chafouleas S.M., Johnson A.J. A review of Generalizability Theory and applications of use in education research: 1976-2010 2013, (under review).
12. Briesch A.M., Chafouleas S.M., Riley-Tillman T.C. Generalizability and dependability of behavioral assessment methods: A comparison of systematic direct observation and Direct Behavior Rating. School Psychology Review 2010, 3:408-421.
13. Cardinet J., Tourneur Y., Allal L. The symmetry of generalizability theory: Applications to educational measurement. Journal of Educational Measurement 1976, 13:119-135.
14. Chafouleas S.M., Briesch A.M., Riley-Tillman T.C., Christ T.J., Black A., Kilgus S.P. An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology 2010, 48:219-246.
15. Chafouleas S.M., Christ T.J., Riley-Tillman T.C., Briesch A.M., Chanese J.A.M. Generalizability and dependability of direct behavior ratings to assess social behavior of preschoolers. School Psychology Review 2007, 35:63-79.
16. Chapman A. The astronomical revolution. Mobius and his band 1993, Oxford University Press, Oxford, England. J. Fauvel, R. Flood, R. Wilson (Eds.).
17. Christ T.C., Ardoin S.P. Curriculum-based measurement of oral reading: Passage equivalence and probe-set development. Journal of School Psychology 2009, 47:55-75.
18. Christ T.J., Johnson-Gros K., Hintze J.M. An examination of computational fluency: The reliability of curriculum-based outcomes within the context of educational decisions. Psychology in the Schools 2005, 42:615-622.
19. Coates T.J., Thoresen C.E. Using generalizability theory in behavioral observations. Behavior Therapy 1978, 9:605-613.
20. Cohen J. Things I have learned (so far). American Psychologist 1990, 45:1304-1312.
21. Cone J.D. The relevance of reliability and validity for behavioral assessment. Behavior Therapy 1977, 8:411-426.
22. Conger A.J., Conger J.C., Wallander J., Ward D., Dygdon J. A generalizability study of the Conners' teacher rating scale-revised. Educational and Psychological Measurement 1983, 43:1019-1031.
23. Crick J.E., Brennan R.L. GENOVA: A generalized analysis of variance system [Computer software and manual] 1983, University of Iowa, Iowa City, IA.
24. Crocker L., Algina J. Introduction to classical and modern test theory 2006, Cengage Learning, Mason, OH.
25. Cronbach L.J. My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement 2004, 64:391-418.
26. Cronbach L.J., Gleser G.C., Nanda H., Rajaratnam N. The dependability of behavioral measurements 1972, Wiley, New York, NY.
27. Cronbach L.J., Rajaratnam N., Gelser G.C. Theory of generalizability: A liberalization of reliability theory. The British Journal of Statistical Psychology 1963, 16:137-163.
28. Crowley S.L., Thompson B., Worchel F. The Children's Depression Inventory: A comparison of generalizability and classical test theory analyses. Educational and Psychological Measurement 1994, 54:705-713.
29. Doros G., Lew R. Design based on intra-class correlation coefficients. American Journal of Biostatistics 2010, 1:1-8.
30. Faul F., Erdfelder E., Lang A.G., Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 2007, 39:175-191.
31. Fisher R.A. Statistical methods for research workers 1938, Hafner, New York, NY. 14th ed.
32. Fitzpatrick A.R., Lee G., Gao F. Assessing the comparability of school scores across test forms that are not parallel. Applied Measurement in Education 2001, 14:285-306.
33. Fuchs L.S., Fuchs D., Hamlett C.L., Walz L., Germann G. Formative evaluation of academic progress: How much growth can we expect?. School Psychology Review 1993, 22:27-48.
34. Gao X., Shavelson R.J., Baxter G.P. Generalizability of large-scale performance assessments in science: Promise and problems. Applied Measurement in Education 1994, 7:323-342.
35. Gillmore G.M., Kane M.T., Naccarato R.W. The generalizability of student ratings of instruction: Estimation of the teacher and course components. Journal of Educational Measurement 1978, 15:1-13.
36. Hammill D.D., Larson S.C. Test of written language-2 1988, PRO-ED, Austin, TX.
37. Hendrickson A., Yin P. Generalizability theory. The reviewer's guide to quantitative methods in the social sciences 2010, Routledge, New York, NY. G.R. Hancock, R.O. Mueller (Eds.).
38. Hintze J.M., Christ T.J., Keller L.A. The generalizability of CBM survey-level mathematics assessments: Just how many samples do we need?. School Psychology Review 2002, 31:514-528.
39. Hintze J.M., Matthews W.J. The generalizability of systematic direct observations across time and setting: a preliminary investigation of the psychometrics of behavioral observation. School Psychology Review 2004, 33:258-270.
40. Hintze J.M., Owen S.V., Shapiro S.V., Daly E.J. Generalizability of oral reading fluency measures: application of G theory to curriculum-based measurement. School Psychology Quarterly 2000, 15:52-68.
41. Hintze J.M., Pelle Pettite H.A. The generalizability of CBM Oral Reading Fluency measures across general and special education. Journal of Psychoeducational Assessment 2001, 19:158-170.
42. Hopkins K.D. Educational and psychological measurement and evaluation 1997, Pearson, Upper Saddle River, NJ. 8th ed.
43. Hoyt C. Test reliability estimated by analysis of variance. Psychometrika 1941, 6:153-160.
44. Hoyt W.T., Kerns M. Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods 1999, 4:403-424. 10.1037/1082-989X.4.4.403.
45. I.B.M.Corporation IBM SPSS statistics for Windows, version 21.0 2012, IBM Corp, Armonk, NY.
46. Jones L.V., Thissen D. A history and overview of psychometrics. Handbook of statistics 2007, Vol. 26:1-27. North Holland, Amsterdam, Netherlands. C.R. Rao, S. Sinharay (Eds.).
47. Kane M.T. A sampling model for validity. Applied Psychological Measurement 1982, 6(2):125-160.
48. Landau S., Stahl D. Sample size and power calculations for medical studies by simulation when closed form expressions are not available. Statistical Methods in Medical Research 2012, 22:324-345.
49. Lee G. The influence of several factors on reliability for complex reading comprehension tests. Journal of Educational Measurement 2002, 39:149-164.
50. Lei P., Smith M., Suen H.K. The use of generalizability theory to estimate data reliability in single-subject observational research. Psychology in the Schools 2007, 44:433-439.
51. Ley P. Quantitative aspects of psychological assessment 1972, Duckworth, London, England.
52. Lomax R.G. An application of generalizability theory to observational research. The Journal of Experimental Education 1982, 51:22-30.
53. Macready G. The use of generalizability theory for assessing relations among items within domains in diagnostic testing. Applied Psychological Measurement 1983, 7:149-157.
54. Martinez J.F., Goldschmidt P., Niemi D., Baker E.L., Sylvester R.M. Language arts performance assignments: Generalizability studies of local and central ratings. Educational Assessment 2007, 12:267-282.
55. Marzano R.J. A comparison of selected methods of scoring classroom assessments. Applied Measurement in Education 2002, 15:249-267.
56. Mastergeorge A.M., Martinez J.F. Rating performance assessments of students with disabilities: A study of reliability and bias. Journal of Psychoeducational Assessment 2009, 28:536-550.
57. McGraw K.O., Wong S.P. Forming inferences about some intraclass correlation coefficients. Psychological Methods 1996, 1:30-46.
58. McWilliam R.A., Ware W.B. The reliability of observations of young children's engagement: An application of generalizability theory. Journal of Early Intervention 1994, 18:34-47.
59. Myung I.J. Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology 2003, 47:90-100.
60. Nelson R.O., Hay L.R., Hay W.M. Comments on Cone's "The relevance of reliability and validity for behavioral assessment". Behavior Therapy 1977, 8:427-430.
61. Novick M.R., Jackson P.H., Thayer D.T. Bayesian inference and the classical test theory model: Reliability and true scores. Psychometrika 1971, 36:261-288.
62. Nunnally J.C., Bernstein I.H. Psychometric theory 1994, McGraw-Hill, New York, NY. 3rd ed.
63. Poncy B.C., Skinner C.H., Axtell P.K. An investigation of the reliability and standard error of measurement of words read correctly per minute using curriculum-based measurement. Journal of Psychoeducational Assessment 2005, 23:326-338.
64. Reynolds C.R., Kamphaus R.W. Behavior assessment system for children 2004, American Guidance Service, Circle Pines, MN. 2nd ed.
65. S.A.S.Corporation SAS 9.2 user's guide 2008, SAS Institute, Cary, NC.
66. Salvia J., Ysselydke J.E., Bolt S. Assessment in special and inclusive education 2010, Houghton Mifflin, Boston, MA. 11th ed.
67. Searle S.R. Notes on variance component estimation. A detailed account of maximum likelihood and kindred methodologies 1979, Cornell University, Ithaca, NY.
68. Shavelson R.J., Webb N.M. Generalizability theory: A primer 1991, Sage, Newbury Park, CA.
69. Shavelson R.J., Webb N.M., Rowley G.L. Generalizability theory. American Psychologist 1989, 44:922-932.
70. Shoukri M.M., Asyali M.H., Donner A. Sample size requirements for the design of reliability study: Review and new results. Statistical Methods in Medical Research 2004, 13:251-271.
71. Shrout P.E., Fleiss J.L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 1979, 86:420-428.
72. Silva F. Psychometric foundations of behavioral assessment 1993, Sage, Newbury Park, CA.
73. Smith P.L. Gaining accuracy in generalizability theory: Using multiple designs. Journal of Educational Measurement 1981, 18:147-154.
74. Solano-Flores G., Li M. The use of generalizability theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice 2006, 25:13-22.
75. Swartz C.W., Hooper S.R., Montgomery J.W., Wakely M.B., de Kruif R.E.L., et al. Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytical scoring methods. Educational and Psychological Measurement 1999, 59:492-506.
76. Swiss Society for Research in Education Working Group EDUG user guide 2006, IRDP, Neuchatel, Switzerland.
77. Thorndike R.L. Reliability. Educational measurement 1951, 560-620. American Council on Education, Washington, DC. E.F. Lindquist (Ed.).
78. Tindal G., Yovanoff P., Geller J.P. Generalizability theory applied to reading assessments for students with significant cognitive disabilities. Journal of Special Education 2010, 44:3-17.
79. Volpe R.J., Briesch A.M., Gadow K.D. The efficiency of behavior rating scales to assess inattentive-overactive and oppositional-defiant behaviors: Applying generalizability theory to streamline assessment. Journal of School Psychology 2011, 49:131-155.
80. Volpe R.J., McConaughy S.H., Hintze J.M. Generalizability of classroom behavior problem and on-task scores from the Direct Observation Form. School Psychology Review 2009, 38:382-401.
81. Walter D.S., Eliasziw M., Donner A. A sample size and optimal design for reliability studies. Statistics in Medicine 1998, 17:101-110.
82. Webb N.M., Rowley G.L., Shavelson R.J. Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development 1988, 21:81-90.
83. Webb N.M., Schlackman J., Sugrue B. The dependability and interchangeability of assessment methods in science. Applied Measurement in Education 2000, 13:277-301.
84. Webb N.M., Shavelson R.J., Maddahian E. Multivariate generalizability theory. New Directions for Testing and Measurement 1983, 18:67-81.
85. Wechsler D. Wechsler intelligence scale for children 2003, Harcourt, San Antonio, TX. 4th ed.
86. Zou G.Y. Sample size formulas for estimating intraclass correalation coefficients with precision and assurance. Statistics in Medicine 2012, 31:3972-3981.