Validation of Multilevel Constructs: Validation Methods and Empirical Findings for the EDI

Social Indicators Research

Volumn 103, Issue 2, 2011, Pages 231-265

  Forer, Barry ^a   Zumbo, Bruno D ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Construct validation; Multilevel; School readiness

Indexed keywords

ANALYTICAL METHOD; EDUCATION; HUMAN DEVELOPMENT INDEX; MEASUREMENT METHOD; METHODOLOGY;

EID: 79961023384     PISSN: 03038300     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11205-011-9844-3     Document Type: Article

References (85)

1. Andrich, D., & Styles, I. (2004). Report on the psychometric properties of the Early Development Instrument (EDI) using the Rasch model. A technical paper commissioned for the development of the Australian Early Development Instrument (AEDI), Murdoch University.
2. Beswick, J. F., Sloat, E. A., & Willms, J. D. (2004). A comparative study of bias in teacher ratings of language and emergent literacy skills. Unpublished manuscript.
3. Blatchford, P., Bassett, P., Goldstein, H., & Martin, C. (2003). Are class size differences related to pupils' educational progress and classroom processes? Findings from the institute of education class size study of children aged 5 to 7 years old. British Educational Research Journal, 29, 709-730.
4. Blatchford, P., Russell, A., Bassett, P., Brown, P., & Martin, C. (2007). The effect of class size on the teaching of pupils aged 7-11 years old. School Effectiveness and School Improvement, 18, 147-172.
5. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
6. Bliese, P. D., & Halverson, R. R. (1998). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157-172.
7. Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305-314.
8. Brinkman, S., Silburn, S., Lawrence, D., Goldfeld, S., Sayers, M., & Oberklaid, F. (2007). Investigating the validity of the Australian Early Development Index. Early Education and Development, 18, 427-451.
9. Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513-531.
10. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Thousand Oaks, CA: Sage.
11. Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum.
12. Carpiano, R. M., Lloyd, J. E. V., & Hertzman, C. (2009). Concentrated affluence, concentrated disadvantage, and children's readiness for school: A population-based, multi-level investigation. Social Science and Medicine, 69, 420-432.
13. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
14. Chen, G., Mathieu, J. E., & Bliese, P. D. (2004a). A framework for conducting multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes. (pp. 273-303). The Netherlands: Elsevier.
15. Chen, G., Mathieu, J. E., & Bliese, P. D. (2004b). Validating frogs and ponds in multi-level contexts: Some afterthoughts. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes. (pp. 335-343). The Netherlands: Elsevier.
16. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage Publications, Inc.
17. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
18. Cronbach, L., & Meehl, P. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.
19. Dansereau, F., Alutto, J. A., & Yammarino, F. J. (1984). Theory testing in organizational behavior: The "varient" approach. Englewood Cliffs, NJ: Prentice-Hall.
20. Dansereau, F., Cho, J., & Yammarino, F. J. (2006). Avoiding the fallacy of the wrong level. Group and Organization Management, 31, 536-577.
21. Dansereau, F., & Yammarino, F. J. (2000). Within and between analysis: The varient paradigm as an underlying approach to theory building and testing. In K. J. Klein & S. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 425-466). San Francisco: Jossey-Bass.
22. Dansereau, F., & Yammarino, F. J. (2004). Overview: Multi-level issues in organizational behavior and processes. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes (pp. Xiii-xxxiii). The Netherlands: Elsevier.
23. Dansereau, F., & Yammarino, F. J. (2006). Is more discussion about levels of analysis really necessary? When is such discussion sufficient? The Leadership Quarterly, 17, 537-552.
24. Diez-Roux, A. V. (1998). Bringing context back into epidemiology: Variables and fallacies in multilevel analysis. American Journal of Public Health, 88, 216-222.
25. Duku, E., & Janus, M. (2004). Stability and reliability of the Early Development Instrument: A population-based measure for communities (EDI). Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.
26. Duncan, G. J., Claessens, A., Huston, A. C., Pagani, L. S., Engel, M., Sexton, H., et al. (2007). School readiness and later achievement. Developmental Psychology, 43, 1428-1446.
27. Ellwein, M. C., Walsh, D. J., Eads, G. M., & Miller, A. (1991). Using readiness tests to route kindergarten students: The snarled intersection of psychometrics, policy, and practice. Educational Evaluation and Policy Analysis, 13, 159-175.
28. Finn, J. D., Pannozzo, G. M., & Achilles, C. M. (2003). The "whys" of class size: Student behavior in small classes. Review of Educational Research, 73, 321-368.
29. Forer, B. (2009). Validation of multilevel constructs: Methods and empirical findings for the Early Development Instrument. Unpublished doctoral dissertation, University of British Columbia.
30. Forget-Dubois, N., Lemelin, J.-P., Boivin, M., Ginette, D., Séguin, J. R., Vitaro, F., et al. (2007). Predicting early school achievement with the EDI: A longitudinal population-based study. Early Education and Development, 18, 405-426.
31. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
32. Griffith, J. (2002). Is quality/effectiveness an empirically demonstrable school attribute? Statistical aids for determining appropriate levels of analysis. School Effectiveness and School Improvement, 13, 91-122.
33. Grilli, L., & Rampichini, C. (2007). Multilevel factor models for ordinal variables. Structural Equation Modeling, 14, 1-25.
34. Guhn, M. & Goelman, H. (2011). Bioecological theory, early child development, and the validation of the population-level Early Development Instrument. Social Indicators Research. doi: 10. 1007/s11205-011-9842-5.
35. Guhn, M., Janus, M., & Hertzman, C. (2007). The Early Development Instrument: Translating school readiness assessment into community actions and policy planning. Early Education and Development, 18, 369-374.
36. Hofmann, D. A., & Jones, L. M. (2004). Some foundational and guiding questions for multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behavior and processes (pp. 305-315). The Netherlands: Elsevier.
37. Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
38. Hu, L. T., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76-99). Thousand Oaks, CA: Sage.
39. Hymel, S., LeMare, L., & McKee, W. (2011). The Early Development Instrument (EDI): An examination of convergent and discriminant validity. Social Indicators Research. doi: 10. 1007/s11205-011-9845-2.
40. Janus, M. (2001). Validation of a teacher measure of school readiness with parent and child-care provider reports. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.
41. Janus, M. (2002). Validation of the Early Development Instrument in a sample of First Nations children. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Canada. Annual Research Day.
42. Janus, M., Brinkman, S., Duku, E., Hertzman, C., Santos, R., Sayers, M., et al. (2007). The Early Development Instrument: A population-based measure for communities. A handbook on development, properties, and use. Hamilton, ON: Offord Centre for Child Studies.
43. Janus, M., Harren, T., & Duku, E. (2004). Neighbourhood perspective on school readiness in kindergarten, academic testing in Grade 3, and affluence levels. Paper presented at the McMaster University Psychiatry Research Day, Hamilton, ON.
44. Janus, M., & Offord, D. (2000). Readiness to learn at school. ISUMA Canadian Journal of Policy Research, 1, 71-75.
45. Janus, M., & Offord, D. (2007). Development and psychometric properties of the Early Development Instrument (EDI): A measure of children's school readiness. Canadian Journal of Behavioural Science, 39, 1-22.
46. Janus, M., Offord, D., & Walsh, C. (2001). Population-level assessment of readiness to learn at school for 5-year-olds in Canada: Relation to child and parent measures. Presented at the Society for Research on Child Development conference, Minneapolis.
47. Janus, M., Walsh, C., & Duku, E. (2005). Early development instrument: Factor structure, sub-domains and Multiple Challenge Index. Department of Psychiatry and Biobehavioural Sciences, McMaster University, Annual Research Day.
48. Janus, M., Walsh, C., Viveiros, H., Duku, E., & Offord, D. (2003). School readiness to learn and neighbourhood characteristics. Poster presented at the Biennial meeting of the Society for Research on Child Development, Tampa, FL.
49. Janus, M., Willms, J. D., & Offord, D. R. (2000). Psychometric properties of the Early Development Instrument (EDI): A teacher-completed measure of children's readiness to learn at school entry. Unpublished manuscript.
50. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education/Praeger.
51. Keating, D. P. (2007). Formative evaluation of the Early Development Instrument: Progress and prospects. Early Education and Development, 18(3), 561-570.
52. Keating, D. P., & Hertzman, C. (Eds.). (1999). Developmental health and the wealth of nations. New York: Guildford.
53. Kershaw, P., & Forer, B. (2006). What are the social and economic indicators of nurturing neighborhoods? Poster presented at the American Educational Research Association conference, San Francisco, California.
54. Kershaw, P., Irwin, L., Trafford, K., & Hertzman, C. (2005). The British Columbia atlas of child development. Vancouver, BC: Human Early Learning Partnership.
55. Kim, K. (2004). An additional view of conducting multi-level construct validation. In F. J. Yammarino & F. Dansereau (Eds.), Multi-level issues in organizational behaviour and processes (pp. 317-333). The Netherlands: Elsevier.
56. Kim, J., & Suen, H. K. (2003). Predicting children's academic achievement from early assessment scores: A validity generalization study. Early Childhood Research Quarterly, 18, 547-566.
57. Klein, K. J., Bliese, P. D., Kozlowski, S. W. J., Dansereau, F., Gavin, M. B., Griffin, M. A., et al. (2000). Multilevel analytic techniques: Commonalities, differences, and continuing questions. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 512-553). San Francisco: Jossey-Bass.
58. Klein, K. J., Dansereau, F., & Hall, R. J. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195-229.
59. Kreiger, J. (2003). Class size reduction: Implementation and solutions. Paper presented at the SERVE Research and Policy Class Size Symposium, Raleigh, SC.
60. LaPointe, V. R. (2006). Conceptualizing and examining the impact of neighbourhoods on the school readiness of kindergarten children in British Columbia. Unpublished Doctoral Dissertation, Vancouver, The University of British Columbia.
61. LaPointe, V. R., Ford, L., & Zumbo, B. D. (2007). Examining the relationship between neighbourhood environment and school readiness for kindergarten children. Early Education and Development, 18, 473-496.
62. Linn, R. L. (2009). The concept of validity in the context of NCLB. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 195-212). Charlotte, NC: IAP-Information Age Publishing, Inc.
63. Lissitz, R. W. (2009). The concept of validity: Revisions, new directions and applications. Charlotte, NC.: IAP-Information Age Publishing, Inc.
64. Lloyd, J. E. V., & Hertzman, C. (2009). From Kindergarten readiness to fourth-grade assessment: Longitudinal analysis with linked population data. Social Science and Medicine, 68, 111-123.
65. Mashburn, A. J., Hamre, B. K., Downer, J. T., & Pianta, R. C. (2006). Teacher and classroom characteristics associated with teachers' ratings of prekindergartners' relationships and behaviors. Journal of Psychoeducational Assessment, 24, 367-380.
66. Meisels, S. J. (1997). Using Work Sampling in authentic performance assessments. Educational Leadership, 54, 60-65.
67. Meisels, S. J. (1999). Assessing readiness. In R. C. Pianta & M. J. Cox (Eds.), The transition to kindergarten (pp. 39-66). Baltimore, MD: Paul H. Brookes.
68. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan.
69. Muthen, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 276-398.
70. Muthen, L. K., & Muthen, B. O. (2006). Mplus User's Guide, version 4. Los Angeles: Muthen and Muthen.
71. National Research Council Institute of Medicine. (2000). From neurons to neighbourhoods: The science of early childhood development. In J. P. Shonkoff & D. A. Phillips (Eds.), Committee on integrating the science of early childhood development (pp. 328-336). Washington, D. C.: National Academy Press.
72. O'Brien, R. M. (1990). Estimating the reliability of aggregate-level variables based on individual-level characteristics. Sociological Methods and Research, 18, 473-504.
73. Ostroff, C. (1992). The relationship between satisfaction, attitudes and performance: An organizational-level analysis. Journal of Applied Psychology, 77, 963-974.
74. Pedder, D. (2006). Are small classes better? Understanding relationships between class size, classroom processes, and pupils' learning. Oxford Review of Education, 32, 213-234.
75. Pellegrini, A., & Blatchford, P. (2000). The child at school: Interactions with peers and teachers. London: Edward Arnold.
76. Province of British Columbia (2008). Strategic plan 2008/09 - 2010/11. Retrieved from http://www. bcbudget. gov. bc. ca/2008/stplan/2008_Strategic_Plan. pdf on September 18, 2008.
77. Rimm-Kaufman, S. E., & Pianta, R. C. (2000). An ecological perspective on the transition to kindergarten: A theoretical framework to guide empirical research. Journal of Applied Developmental Psychology, 21, 491-511.
78. Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351-357.
79. Rowan, B., Raudenbush, S. W., & Kang, S. K. (1991). School climate in secondary schools. In S. W. Raudenbush & J. D. Willms (Eds.), Schools, classrooms, and pupils: International studies of schooling from a multilevel perspective (pp. 203-223). San Diego: Academic.
80. Satorra, A., & Bentler, P. M. (1999). A scaled difference Chi-square test statistic for moment structure analysis. Technical Report, University of California, Los Angeles. http://preprints. stat. ucla. edu/260/chisquare. pdf.
81. Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Routledge.
82. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Cambridge, MA: Harvard University Press.
83. Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics, Vol. 26: Psychometrics (pp. 45-79). Elsevier Science B. V.: The Netherlands.
84. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65-82). Charlotte, NC: IAP-Information Age Publishing, Inc.
85. Zumbo, B. D. & Forer, B. (2011). Testing and measurement from a multilevel view: Psychometrics and validation. In J. Bovaird, K. Geisinger, & C. Buckendahl (Eds.). High stakes testing in education-science and practice in K-12 settings (Festschrift to Barbara Plake). American Psychological Press: Washington, D. C.