Inter-rater and test-retest reliability: Methods and results for the neighborhood observational checklist

Health and Place

Volumn 13, Issue 2, 2007, Pages 452-465

  Zenk, Shannon N ^a   Schulz, Amy J ^b   Mentz, Graciela ^b   House, James S ^c   Gravlee, Clarence C ^d   Miranda, Patricia Y ^b   Miller, Patricia ^e   Kannan, Srimathi ^b  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^b UNIVERSITY OF MICHIGAN SCHOOL OF PUBLIC HEALTH   (United States)

^c UNIVERSITY OF MICHIGAN   (United States)

^d FLORIDA STATE UNIVERSITY   (United States)

^e Southwest Counseling and Development Services   (United States)

Author keywords

Direct observation; Environmental audit; Neighborhood; Reliability; Systematic social observation; Urban health

Indexed keywords

ENVIRONMENTAL AUDITING; NEIGHBORHOOD; PUBLIC HEALTH; RELIABILITY ANALYSIS; RESEARCH;

ARTICLE; CHECKLIST; DATA ANALYSIS; EVALUATION; INFORMATION PROCESSING; INSTRUMENTATION; KAPLAN MEIER METHOD; MEDICAL RESEARCH; METHODOLOGY; OBSERVATION; PRIORITY JOURNAL; RELIABILITY; UNITED STATES;

DETROIT; MICHIGAN; NORTH AMERICA; UNITED STATES;

EID: 33846399358     PISSN: 13538292     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.healthplace.2006.05.003     Document Type: Article

References (37)

1. Brownson R.C., Hoehner C.M., Brennan L.K., Cook R.A., Elliott M.B., and McMullen K. Reliability of two instruments for auditing the environment for physical activity. Journal of Physical Activity and Health 1 (2004) 191-208
2. Carmines E.G., and Zeller R.A. Reliability and Validity Assessment (1979), Sage, Newbury Park
3. Caughy M.O., O'Campo P.J., and Patterson J. A brief observational measure for urban neighborhoods. Health & Place 7 (2001) 225-236
4. Cohen D.A., Mason K., Bedimo A., Scribner R., Basolo V., and Farley T.A. Neighborhood physical conditions and health. American Journal of Public Health 93 (2003) 467-471
5. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1960) 37-46
6. Day K., Boarnet M., Alfonzo M., and Forsyth A. The Irvine-Minnesota inventory to measure built environments: Development. American Journal of Preventive Medicine 30 (2006) 144-152
7. Diez Roux A.V. Residential environments and cardiovascular risk. Journal of Urban Health 80 (2003) 569-589
8. Ellen I.G., Mijanovich T., and Dillman K.N. Neighborhood effects on health: exploring the links and assessing the evidence. Journal of Urban Affairs 23 (2001) 391-408
9. Farquhar S.A. Effects of the perceptions and observations of environmental stressors on health and well-being in residents of eastside and southwest Detroit. Health Behavior and Health Education (2000), University of Michigan, Ann Arbor
10. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychological Bulletin 76 (1971) 378-382
11. Forsyth D.R. Group Dynamics (1999), Wadsworth, Belmont, CA
12. Gravlee, C.C., Zenk, S.N., Woods, S., Rowe, Z., Schulz, A.. Handheld computers for direct observation of the social and physical environment. Field Methods, in press.
13. Gwet K. Computing inter-rater reliability with the SAS system. Statistical Methods for Inter-rater Reliability Assessment 3 (2002) 1-16
14. Gwet K. Inter-rater reliability: dependency on trait prevalence and marginal homogeneity. Statistical Methods for Inter-rater Reliability Assessment 2 (2002) 1-9
15. Israel B.A., Schulz A.J., Estrada-Martinez L., Zenk S.N., Viruell-Fuentes E., Villarruel A.M., and Stokes C. Engaging urban residents in assessing neighborhood environments and their implications for health. Journal of Urban Health 83 (2006) 523-539
16. Jencks C., and Mayer S.E. The social consequences of growing up in a poor neighborhood. In: Lynn L.E., and McGeary M.G.H. (Eds). Inner City Poverty in the United States (1990), National Academy, Washington DC 111-186
17. Kendall M.G. Rank Correlation Methods. second ed (1955), Charles Griffin & Co, London
18. Kobak K.A., Engelhardt N., Williams J.B.W., and Lipsitz J.D. Rater training in multicenter clinical trials: issues and recommendations. Journal of Clinical Psychopharmacology 24 (2004) 113-117
19. Landis J.R., and Koch G. The measurement of observer agreement for categorical data. Biometrics 33 (1977) 159-174
20. Lee J., and Wong D.W. Statistical Analysis with ArcView GIS (2001), Wiley, New York
21. Levy S.R., Anderson E.E., Issel L.M., Willis M.A., Dancy B.L., et al. Using multilevel, multisource needs assessment data for planning community interventions. Health Promotion Practice 5 (2004) 59-68
22. Macintyre S., Ellaway A., and Cummins S. Place effects on health: how can we conceptualise, operationalise and measure them?. Social Science & Medicine 55 (2002) 125-139
23. Morenoff, J., House, J.S., Raudenbush, S. Systematic social observation by survey interviewers: A methodological evaluation, in preparation.
24. Mulsant B.H., Kastango K.B., Rosen J., Stone R.A., Mazumdar S., and Pollock B.G. Interrater reliability in clinical trials of depressive disorders. American Journal of Psychiatry 159 (2002) 1598-1600
25. Perkins D.D., Meeks J.W., and Taylor R.B. The physical environment of street blocks and resident perceptions of crime and disorder: implications for theory and measurement. Journal of Environmental Psychology 12 (1992) 21-34
26. Pikora T.J., Bull F.C.L., Jamrozik K., Knuiman M., Giles-Corti B., and Donovan R.J. Developing a reliable audit instrument to measure the physical environment for physical activity. American Journal of Preventive Medicine 23 (2002) 187-194
27. Raudenbush S.W., and Sampson R.J. Ecometrics: toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Sociological Methodology 29 (1999) 1-41
28. Robert S.A. Socioeconomic position and health: the independent contribution of community socioeconomic context. Annual Review of Sociology 25 (1999) 489-516
29. Ross C.E., and Mirowsky J. Neighborhood disadvantage, disorder, and health. Journal of Health and Social Behavior 42 (2001) 258-276
30. Sampson R.J., and Groves W.B. Community structure and crime: testing social disorganization theory. American Journal of Sociology 94 (1989) 774-802
31. Sampson R.J., and Raudenbush S.W. Systematic social observation of public spaces: a new look at disorder in urban neighborhoods. American Journal of Sociology 105 (1999) 603-651
32. Sampson R.J., and Raudenbush S.W. Seeing disorder: Neighborhood stigma and the social construction of "broken windows". Social Psychology Quarterly 67 (2004) 319-342
33. Sastry, N., Pebley, A.R., 2003. The Los Angeles family and neighborhood survey: neighborhood observation forms and interviewer manual. RAND Working Paper DRU-2400/6-LAFANS.
34. Schulz A.J., Kannan S., Dvonch J.T., Israel B.A., Allen A., James S.A., House J.S., and Lepkowski J. Social and physical environments and disparities in risk for cardiovascular disease: the Healthy Environments Partnership conceptual model. Environmental Health Perspectives 113 (2005) 1817-1825
35. Weich S., Burton E., Blanchard M., Prince M., Sproston K., and Erens B. Measuring the built environment: validity of a site survey instrument for use in urban settings. Health & Place 7 (2001) 283-292
36. Wilson J.Q., and Kelling G.L. Broken windows. Atlantic Monthly 249 (1982) 29-38
37. Zenk, S.N., Schulz, A.J., House, J. S., Benjamin, A., Kannan, S., 2005. Application of community-based participatory research in the design of an observational tool: the neighborhood observational checklist. In: Israel, B.A., Eng, E., Schulz, A.J., Parker, E.A. (Eds.), Methods in Community-Based Participatory Research for Health. San Francisco, Jossey-Bass, pp. 167-187.