Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments

Journal of Personality Assessment

Volumn 100, Issue 1, 2018, Pages 53-67

  Vispoel, Walter P ^a   Morris, Carrie A ^a   Kilinc, Murat ^a  

^a UNIVERSITY OF IOWA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION COEFFICIENT; HUMAN; HUMAN EXPERIMENT; MEASUREMENT ERROR; PSYCHOLOGIC ASSESSMENT; QUANTITATIVE STUDY; SOFTWARE; STRUCTURAL EQUATION MODELING; THEORETICAL MODEL; PERSONALITY ASSESSMENT; PSYCHOLOGIC TEST; PSYCHOLOGICAL MODEL; REPRODUCIBILITY; STANDARDS;

HUMANS; MODELS, PSYCHOLOGICAL; PERSONALITY ASSESSMENT; PSYCHOLOGICAL TESTS; REPRODUCIBILITY OF RESULTS;

EID: 85017580919     PISSN: 00223891     EISSN: None     Source Type: Journal    
DOI: 10.1080/00223891.2017.1296455     Document Type: Article

References (53)

1. Ark, T. K., (2015). Ordinal generalizability theory using an underlying latent variable framework (Doctoral dissertation). Retrieved from https://open.library.ubc.ca/cIRcle/collections/ubctheses/24/items/1.0166304
2. Bates, D., Maechler, M., Bolker, B., & Walker, S., (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01
3. Becker, G., (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370–379. doi:10.1037/1082-989X.5.3.370
4. Brennan, R. L., (2001). Generalizability theory. New York, NY: Springer-Verlag.
5. Brennan, R. L., & Kane, M. T., (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. doi:10.1111/j.1745-3984.1977.tb00045.x
6. Brown, W., (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322. doi:10.1111/j.2044-8295.1910.tb00207.x
7. Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. New York, NY: Routledge.
8. Chmielewski, M., & Watson, D., (2009). What is being assessed and why it matters: The impact of transient error in trait research [Personality processes and individual differences]. Journal of Personality and Social Psychology, 97, 186–202. doi:10.1037/a0015618
9. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N., (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
10. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston.
11. Cronbach, L. J., Rajaratnam, N., & Gleser, G. C., (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. doi:10.1111/j.2044-8317.1963.tb00206.x
12. de Vries, R. E., Zettler, I., & Hilbig, B. E., (2014). Rethinking trait conceptions of social desirability scales: Impression management as an expression of honesty-humility. Assessment, 21, 286–299. doi:10.1177/1073191113504619
13. Dunn, T. J., Baguley, T., & Brunsden, V., (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399–412. doi:10.1111/bjop.12046
14. Geiser, C., Keller, B. T., Lockhart, G., Eid, M., Cole, D. A., & Koch, T., (2015). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behavior Research Methods, 47, 172–203. doi:10.3758/s13428-014-0457-z
15. Gleser, G. C., Cronbach¸ L. J., & Rajaratnam, N., (1965). Generalizability of scores influenced by multiple sources of variance. Psychometrika, 30, 395–418. doi:10.1007/BF02289531
16. Graham, J. M., (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability. Educational and Psychological Measurement, 66, 930–944. doi:10.1177/0013164406288165
17. Green, S. B., & Yang, Y., (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121–135. doi:10.1007/s11336-008-9098-4
18. Green, S. B., Yang, Y., Alt, M., Brinkley, S., Gray, S., Hogan, T., & Cowan, N., (2016). Use of internal consistency coefficients for estimating reliability of experimental tasks. Psychonomic Bulletin and Review, 23, 750–763. doi:10.3758/s13423-015-0968-3
19. Hall, R. J., Snell, A. F., & Foust, M. S., (1999). Item parceling strategies in SEM: Investigating the subtle effects of unmodeled secondary constructs. Organizational Research Methods, 2, 233–256.
20. Huang, C., (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28, 663–683. doi:10.1007/s10212-012-0134-5
21. Le, H., Schmidt, F. L., & Putka, D. J., (2009). The multifaceted nature of measurement artifacts and its implications for estimating construct-level relationships. Organizational Research Methods, 12, 165–200. doi:10.1177/1094428107302900
22. Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F., (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151–173. doi:10.1207/S15328007SEM0902_1
23. Little, T. D., Rhemtulla, M., Gibson, K., & Schoemann, A. M., (2013). Why the items versus parcels controversy needn't be one. Psychological Methods, 18, 285–300. doi:10.1037/a0033266
24. Marcoulides, G. A., (1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 379–386. doi:10.2466/pr0.1990.66.2.379
25. Marcoulides, G. A., (2000, March). Generalizability theory: Advancements and implementations. Invited colloquium presented at the 22nd Language Testing Research Colloquium, Vancouver, BC, Canada.
26. Marsh, H. W., (1992). Self-Description Questionnaire (SDQ) III: A theoretical and empirical basis for the measurement of multiple dimensions of late adolescent self-concept. An interim test manual and research monograph. Macarthur, New South Wales, Australia: University of Western Sydney, Faculty of Education.
27. Marsh, H. W., Lüdtke, O., Nagengast, B., Morin, A. J., & Von Davier, M., (2013). Why item parcels are (almost) never appropriate: Two wrongs do not make a right—Camouflaging misspecification with item parcels in CFA models. Psychological Methods, 18, 257–284. doi:10.1037/a0032773
28. McCrae, R. R., (2015). A more nuanced view of reliability: Specificity in the trait hierarchy. Personality and Social Psychology Review, 19, 97–112. doi:10.1177/1088868314541857
29. McCrae, R. R., & Costa, P. T., Jr. (2010). NEO Inventories professional manual. Odessa, FL: Psychological Assessment Resources.
30. McCrae, R. R., Kurtz, J. E., Yamagata, S., & Terracciano, A., (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28–50. doi:10.1177/1088868310366253
31. McDonald, R. P., (1999). Test theory: A unified approach. Mahwah, NJ: Erlbaum.
32. Paulhus, D. L., (1991). Measurement and control of response bias. In J. P., Robinson, P. R., Shaver, & L. S., Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.
33. Paulhus, D. L., & Trapnell, P. D., (2008). Self-presentation: An agency–communion framework. In O. P., John, R. W., Robins, & L. A., Pervin (Eds.), Handbook of personality psychology: Theory and research (3rd ed., pp. 492–517). New York, NY: Guilford Press.
34. Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & the R Core Team. (2016). nlme: Linear and nonlinear mixed effects models (3.1–128) [Software package and manual]. Retrieved from https://cran.r-project.org/web/package=nlme
35. Rajaratnam, N., Cronbach, L. J., & Gleser, G. C., (1965). Generalizability of stratified-parallel tests. Psychometrika, 30, 39–56. doi:10.1007/BF02289746
36. Raju, N. S., (1977). A generalization of coefficient alpha. Psychometrika, 42, 549–565. doi:10.1007/BF02295978
37. Reeve, C. L., Heggestad, E. D., & George, E., (2005). Estimation of transient error in cognitive ability scales. International Journal of Selection and Assessment, 13, 316–320. doi:10.1111/j.1468-2389.2005.00328.x
38. Revelle, W., (2016). psych: Procedures for personality and psychological research (1.6.4) [Software package and manual]. Evanston, IL: Northwestern University. Retrieved from https://cran.r-project.org/web/package=psych
39. Rosseel, Y., (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. doi:10.18637/jss.v048.i02
40. Rulon, P. J., (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9, 99–103.
41. Schmidt, F. L., Le, H., & Ilies, R., (2003). Beyond alpha: An empirical investigation of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206–224. doi:10.1037/1082-989X.8.2.206
42. Sijtsma, K., (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107–120. doi:10.1007/s11336-008-9101-0
43. Spearman, C., (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101. doi:10.1093/ije/dyq191
44. Spearman, C., (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295. doi:10.1111/j.2044-8295.1910.tb00206.x
45. Steyer, R., Ferring, D., & Schmitt, M. J., (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79–98.
46. Steyer, R., Mayer, A., Geiser, C., & Cole, D. A., (2015). A theory of states and traits—revised. Annual Review of Clinical Psychology, 11, 71–98. doi:10.1146/annurev-clinpsy-032813-153719
47. Thorndike, R. L., (1951). Reliability. In E. F., Lindquist (Ed.), Educational measurement (pp. 560–620). Washington, DC: American Council on Education. Retrieved from https://archive.org/details/educationalmeasu00lind
48. Vispoel, W. P., & Forte, Fast, E. E., (2000). Response biases and their relation to sex differences in multiple domains of self-concept. Applied Measurement in Education, 13, 79–97. doi:10.1207/s15324818ame1301_4
49. Vispoel, W. P., & Kim, H. Y., (2014). Psychometric properties for the Balanced Inventory of Desirable Responding: Dichotomous versus polytomous conventional and IRT scoring. Psychological Assessment, 26, 878–891. doi:10.1037/a0036430
50. Vispoel, W. P., Morris, C. A., & Kilinc, M., (2016). Using G-theory to enhance evidence of reliability and validity for common uses of the Paulhus Deception Scales. Assessment. Advance online publication. doi:10.1177/1073191116641182
51. Vispoel, W. P., Morris, C. A., & Kilinc, M., (2017). Applications of generalizability theory and their relations to classical test theory and structural equation modeling. Psychological Methods. Advance online publication. doi:10.1037/met0000107
52. Vispoel, W. P., & Tao, S., (2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94–104. doi:10.1037/a0029061
53. H: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. doi:10.1007/s11336-003-0974-7