Applications of generalizability theory and their relations to classical test theory and structural equation modeling

Psychological Methods

Volumn 23, Issue 1, 2018, Pages 1-26

  Vispoel, Walter P ^a   Morris, Carrie A ^a   Kilinc, Murat ^a  

^a UNIVERSITY OF IOWA   (United States)

Author keywords

Classical Test Theory; Generalizability Theory; Reliability; Structural Equation Modeling; Validity

Indexed keywords

DECISION MAKING; MEASUREMENT ERROR; PSYCINFO; QUANTITATIVE STUDY; RELIABILITY; STRUCTURAL EQUATION MODELING; THEORETICAL MODEL; VALIDITY; HUMAN; METHODOLOGY; PROCEDURES; PSYCHOLOGICAL THEORY; PSYCHOLOGY; PSYCHOMETRY;

HUMANS; MODELS, THEORETICAL; PSYCHOLOGICAL THEORY; PSYCHOLOGY; PSYCHOMETRICS; RESEARCH DESIGN;

EID: 85009968832     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000107     Document Type: Article

References (71)

1. Ark, T. K. (2015, June 11). Ordinal generalizability theory using an underlying latent variable framework (Unpublished doctoral dissertation). University of British Columbia, Vancouver, Canada. http://dx.doi.org/10.14288/1.0166304
2. Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370-379. http://dx.doi.org/10.1037/1082-989X.5.3.370
3. Brennan, R. L. (1998). Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement, 22, 307-331. http://dx.doi.org/10.1177/014662169802200401
4. Brennan, R. L. (2001a). Generalizability theory. New York, NY: Springer-Verlag. http://dx.doi.org/10.1007/978-1-4757-3456-0
5. Brennan, R. L. (2001b). Manual for mGENOVA (Version 2.1). Iowa City, IA: University of Iowa, IA Testing Programs. (Iowa Testing Programs Occasional Papers No. 50)
6. Brennan, R. L. (2001c). Manual for urGENOVA (Version 2.1). Iowa City, IA: University of Iowa, IA Testing Programs.
7. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277-289. http://dx.doi.org/10.1111/j.1745-3984.1977.tb00045.x
8. Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322. http://dx.doi.org/10.1111/j.2044-8295.1910.tb00207.x
9. Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. New York, NY: Routledge.
10. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston.
11. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. http://dx.doi.org/10.1007/BF02310555
12. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
13. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability of performance assessments of school achievement and school effectiveness. Educational and Psychological Measurement, 57, 373-399. http://dx.doi.org/10.1177/0013164497057003001
14. Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163. http://dx.doi.org/10.1111/j.2044-8317.1963.tb00206.x
15. Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418. http://dx.doi.org/10.1177/0013164404266386
16. de Finetti, B. (1937). La prévision: Ses lois logiques, ses sources subjectives [Forecast/prediction: Its logical laws, its subjective sources]. Annales de linstitut Henri Poincaré, 7, 1-68. Retrieved from https://eudml.org/doc/79004 (Published in English in Studies in SubjectiveProbability, edited by H. E. Kyburg Jr., & H. Smokler, New York, NY, Wiley, 1964, pp. 95-158).
17. Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412. http://dx.doi.org/10.1111/bjop.12046
18. Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., & Lischetzke, T. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253. http://dx.doi.org/10.1037/a0013219
19. Feldt, L. S. (1984). Some relationships between the binomial error model and classical test theory. Educational and Psychological Measurement, 44, 883-891. http://dx.doi.org/10.1177/0013164484444010
20. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. H. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York, NY: Macmillan.
21. Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-Type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17, 1-13.
22. Geiser, C., Keller, B. T., Lockhart, G., Eid, M., Cole, D. A., & Koch, T. (2015). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-Trait analyses. Behavior Research Methods, 47, 172-203. http://dx.doi.org/10.3758/s13428-014-0457-z
23. Green, S. B. (2003). A coefficient alpha for test-retest data. Psychological Methods, 8, 88-101. http://dx.doi.org/10.1037/1082-989X.8.1.88
24. Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121-135. http://dx.doi.org/10.1007/s11336-008-9098-4
25. Haertel, E. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: Praeger.
26. Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72, 320-338. http://dx.doi.org/10.1080/01621459.1977.10480998
27. Helmreich, R., & Stapp, J. (1974). Short forms of the Texas Social Behavior Inventory (TSBI), an objective measure of self-esteem. Bulletin of the Psychonomic Society, 4, 473-475. http://dx.doi.org/10.3758/BF03334260
28. Huang, C. (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28, 663-683. http://dx.doi.org/10.1007/s10212-012-0134-5
29. Jarjoura, D. (1986). An estimator of examinee-level measurement error variance that considers test form difficulty adjustments. Applied Psychological Measurement, 10, 175-186. http://dx.doi.org/10.1177/014662168601000209
30. John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five Inventory-Versions 4a and 54. Berkeley, CA: University of California, Berkeley, Institute of Personality and Social Research. Retrieved from http://www.outofservice.com/bigfive
31. Le, H., Schmidt, F. L., & Putka, D. J. (2009). The multifaceted nature of measurement artifacts and its implications for estimating construct-level relationships. Organizational Research Methods, 12, 165-200. http://dx.doi.org/10.1177/1094428107302900
32. Livingston, S. A. (1972). Criterion-referenced applications of classical test theory. Journal of Educational Measurement, 9, 13-26. http://dx.doi.org/10.1111/j.1745-3984.1972.tb00756.x
33. LoPilato, A. C., Carter, N. T., & Wang, M. (2015). Updating generalizability theory in management research: Bayesian estimation of variance components. Journal of Management, 41, 692-717. http://dx.doi.org/10.1177/0149206314554215
34. Lord, F. M. (1957). A significance test for the hypothesis that two variables measure the same trait except for errors of measurement. Psychometrika, 22, 207-220. http://dx.doi.org/10.1007/BF02289122
35. Lord, F. M. (1965). A strong true-score theory, with applications. Psychometrika, 30, 239-270. http://dx.doi.org/10.1007/BF02289490
36. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
37. Marcoulides, G. A. (1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 379-386. http://dx.doi.org/10.2466/pr0.1990.66.2.379
38. Marcoulides, G. A. (1996). Estimating variance components in generalizability theory: The covariance structure analysis approach [Teachers corner]. Structural Equation Modeling, 3, 290-299. http://dx.doi.org/10.1080/10705519609540045
39. Marcoulides, G. A. (2000). Generalizability theory. In H. Tinsley & S. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 527-551). San Diego, CA: Academic Press. http://dx.doi.org/10.1016/B978-012691360-6/50019-7
40. McDonald, R. P. (1999). Test theory: A unified approach. Mahwah, NJ: Erlbaum.
41. Mushquash, C., & OConnor, B. P. (2006). SPSS and SAS programs for generalizability theory analyses. Behavior Research Methods, 38, 542-547. http://dx.doi.org/10.3758/BF03192810
42. Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17-59). San Diego, CA: Academic Press. http://dx.doi.org/10.1016/B978-0-12-590241-0.50006-X
43. Paulhus, D. L. (1999). Paulhus Deception Scales (PDS): The Balanced Inventory of Desirable Responding-7 (Users manual). Toronto, Ontario, Canada: Multi-Health Systems.
44. Paulhus, D. L., & Reid, D. B. (1991). Enhancement and denial in socially desirable responding. Journal of Personality and Social Psychology, 60, 307-317. http://dx.doi.org/10.1037/0022-3514.60.2.307
45. Raju, N. S. (1977). A generalization of coefficient alpha. Psychometrika, 42, 549-565. http://dx.doi.org/10.1007/BF02295978
46. Raykov, T., & Marcoulides, G. A. (2006). Estimation of generalizability coefficients via a structural equation modeling approach to scale reliability evaluation. International Journal of Testing, 6, 81-95. http://dx.doi.org/10.1207/s15327574ijt0601-5
47. Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York, NY: Routledge.
48. Revelle, W. (2015). psych: Procedures for personality and psychological research (1.5.8) [Computer software package and manual]. Evanston, IL: Northwestern University. Retrieved from https://cran.r-project.org/web/packages/psych
49. Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. http://dx.doi.org/10.1037/a0029315
50. Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9, 99-103.
51. Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206-224. http://dx.doi.org/10.1037/1082-989X.8.2.206
52. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353. http://dx.doi.org/10.1037/1040-3590.8.4.350
53. Searle, S. R. (1971). Linear models. New York, NY: Wiley.
54. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: Sage.
55. Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932. http://dx.doi.org/10.1037/0003-066X.44.6.922
56. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbachs alpha. Psychometrika, 74, 107-120. http://dx.doi.org/10.1007/s11336-008-9101-0
57. Smith, P. L. (1978). Sampling error of variance components in small sample multifacet generalizability designs. Journal of Educational Measurement, 3, 319-346.
58. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101. (Reprinted in International Journal of Epidemiology, 39, pp. 1137-1150, 2010).http://dx.doi.org/10.1093/ije/dyq191
59. Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271-295. http://dx.doi.org/10.1111/j.2044-8295.1910.tb00206.x
60. Steyer, R., Ferring, D., & Schmitt, M. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79-98.
61. Steyer, R., Mayer, A., Geiser, C., & Cole, D. A. (2015). A theory of states and traits-Revised. Annual Review of Clinical Psychology, 11, 71-98. http://dx.doi.org/10.1146/annurev-clinpsy-032813-153719
62. Stöber, J., Dette, D. E., & Musch, J. (2002). Comparing continuous and dichotomous scoring of the balanced inventory of desirable responding. Journal of Personality Assessment, 78, 370-389. http://dx.doi.org/10.1207/S15327752JPA7802-10
63. Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement (pp. 560-620). Washington, DC: American Council on Education.
64. Vispoel, W. P., & Kim, H. Y. (2014). Psychometric properties for the Balanced Inventory of Desirable Responding: Dichotomous versus polytomous conventional and IRT scoring. Psychological Assessment, 26, 878-891. http://dx.doi.org/10.1037/a0036430
65. Vispoel, W. P., Morris, C. A., & Kilinc, M. (2016). Using G-Theory to enhance evidence of reliability and validity for common uses of the Paulhus Deception Scales. Assessment. Advance online publication. http://dx.doi.org/10.1177/1073191116641182
66. Vispoel, W. P., & Tao, S. (2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94-104. http://dx.doi.org/10.1037/a0029061
67. Wiley, E. W., Webb, N. M., & Shavelson, R. J. (2013). The generalizability of test scores. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 43-60). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/14047-003
68. Woodruff, D. (1991). Stepping up test score conditional variances. Journal of Educational Measurement, 28, 191-196. http://dx.doi.org/10.1111/j.1745-3984.1991.tb00353.x
69. Woodruff, D., Traynor, A., Cui, Z., & Fang, Y. (2013, October). A comparison of three methods for computing scale score conditional standard errors of measurement (ACT Research Report Series 7). Iowa City, IA: ACT.
70. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbachs , Revelle , and McDonalds H: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123-133. http://dx.doi.org/10.1007/s11336-003-0974-7
71. Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods: JMASM, 6, 21-29. Retrieved from http://digitalcommons.wayne.edu/jmasm