Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research

Journal of Planning Education and Research

Volumn 34, Issue 1, 2014, Pages 77-93

  Stevens, Mark R ^a   Lyles, Ward ^b   Berke, Philip R ^c  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF KANSAS   (United States)

^c UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

content analysis; intercoder reliability; Krippendorff's alpha; percent agreement; plan evaluation; plan quality

Indexed keywords

EID: 84893864907     PISSN: 0739456X     EISSN: None     Source Type: Journal    
DOI: 10.1177/0739456X13513614     Document Type: Article

References (70)

1. Aytur Semra A.,Satinsky Sara B.,Evenson Kelly R.,Rodriguez Daniel A.Pedestrian and Bicycle Planning in Rural Communities: Tools for Active Living.Family & Community Health. 2011;34 (2): 173-81.
2. Baer William C.General Plan Evaluation Criteria: An Approach to Making Better Plans.Journal of the American Planning Association. 1997;63 (3): 329-44.
3. Baker Ingrid,Peterson Ann,Brown Greg,McAlpine Clive.Local Government Response to the Impacts of Climate Change: An Evaluation of Local Climate Adaptation Plans.Landscape and Urban Planning. 2012;107 (2): 127-36.
4. Bassett Ellen,Shandas Vivek.Innovation and Climate Action Planning-Perspectives from Municipal Plans.Journal of the American Planning Association. 2010;76 (4): 435-50.
5. Baynham Maggie,Stevens Mark.Are We Planning Effectively for Climate Change? An Evaluation of Official Community Plans in British Columbia.Journal of Environmental Planning and Management. 2013;:.
6. Berke Philip,French Steven P.The Influence of State Planning Mandates on Local Plan Quality.Journal of Planning Education and Research. 1994;13 (4): 237-50.
7. Berke Philip,Godschalk David.Searching for the Good Plan: A Meta-analysis of Plan Quality Studies.Journal of Planning Literature. 2009;23 (3): 227-40.
8. Berke Philip R.Evaluating Environmental Plan Quality: The Case of Planning for Sustainable Development in New Zealand.Journal of Environmental Planning and Management. 1994;37 (2): 155-70.
9. Berke Philip R.,Crawford Jan,Dixon Jenny,Ericksen Neil.Do Cooperative Environmental Planning Mandates Produce Good Plans? Empirical Results from the New Zealand Experience.Environment and Planning B: Planning and Design. 1999;26 (5): 643-64.
10. Berke Philip R.,Dixon Jenny,Ericksen Neil.Coercive and Cooperative Intergovernmental Mandates: A Comparative Analysis of Florida and New Zealand Environmental Plans.Environment and Planning B: Planning and Design. 1997;24 (3): 451-68.
11. Berke Philip R.,Ericksen Neil,Crawford Jan,Dixon Jenny.Planning for Indigenous People: Human Rights and Environmental Protection in New Zealand.Journal of Planning Education and Research. 2002;22 (2): 115-34.
12. Berke Philip R.,Godschalk David R.,Kaiser Edward J.Rodriguez Daniel A.Urban Land Use Planning. 2006. Urban Land Use Planning, 5th ed.Urbana: University of Illinois Press.Urbana: University of Illinois Press; 2006:.
13. Berke Philip R.,Manta-Conroy Maria.Are We Planning for Sustainable Development? An Evaluation of 30 Comprehensive Plans.Journal of the American Planning Association. 2000;66 (1): 21-33.
14. Berke Philip R.,Roenigk Dale J.,Kaiser Edward J.,Burby Raymond.Enhancing Plan Quality: Evaluating the Role of State Planning Mandates for Natural Hazard Mitigation.Journal of Environmental Planning & Management. 1996;39 (1): 79-96.
15. Berke Philip,Smith Gavin,Lyles Ward.Planning for Resiliency: Evaluation of State Hazard Mitigation Plans under the Disaster Mitigation Act.Natural Hazards Review. 2012;13 (2): 139-49.
16. Berke Philip,Spurlock Danielle,Hess George,Band Larry.Local Comprehensive Plan Quality and Regional Ecosystem Protection: The Case of the Jordan Lake Watershed, North Carolina, U.S.A.Land Use Policy. 2013;31:450-9.
17. Brody Samuel D.Are We Learning to Make Better Plans? A Longitudinal Analysis of Plan Quality Associated with Natural Hazards.Journal of Planning Education and Research. 2003a;23 (2): 191-201.
18. Brody Samuel D.Implementing the Principles of Ecosystem Management through Local Land Use Planning.Population and Environment. 2003b;24 (6): 511-40.
19. Brody Samuel D.,Carrasco Virginia,Highfield Wesley E.Measuring the Adoption of Local Sprawl Reduction Planning Policies in Florida.Journal of Planning Education and Research. 2006;25 (3): 294-310.
20. Brody Samuel D.,Highfield Wes,Carrasco Virginia.Measuring the Collective Planning Capabilities of Local Jurisdictions to Manage Ecological Systems in Southern Florida.Landscape and Urban Planning. 2004;69 (1): 33-50.
21. Bunnell Gene,Jepson Edwad J.The Effect of Mandated Planning on Plan Quality.Journal of the American Planning Association. 2011;77 (4): 338-53.
22. Burby Raymond J.,Dalton Linda C.Plans Can Matter! The Role of Land Use Plans and State Planning Mandates in Limiting the Development of Hazardous Areas.Public Administration Review. 1994;54 (3): 229-38.
23. Burby Raymond J.,May Peter J.Berke Philip R.,Dalton Linda C.,French Steven P.,Kaiser Edward J.Making Governments Plan: State Experiments in Managing Land Use. Baltimore: Johns Hopkins University Press; 1997:.
24. Carmines Edward G.,Zeller Richard A.Reliability and Validity Assessment. Newbury Park, CA: Sage; 1979:.
25. Cohen Jacob.A Coefficient of Agreement for Nominal Scales.Educational and Psychological Measurement. 1960;20 (1): 37-46.
26. Compton D'Lane,Love Tony P.,Sell Jane.Developing and Assessing Intercoder Reliability in Studies of Group Interaction.Sociological Methodology. 2012;42 (1): 348-64.
27. Dalton Linda C.,Burby Raymond J.Mandates, Plans, and Planners: Building Local Commitment to Development Management.Journal of the American Planning Association. 1994;60 (4): 444-61.
28. Davis Braxton C.Regional Planning in the US Coastal Zone: A Comparative Analysis of 15 Special Area Plans.Ocean and Coastal Management. 2004;47 (1-2): 79-94.
29. Deyle Robert E.,Smith Richard A.Local Government Compliance with State Plan Mandates: The Effects of State Implementation in Florida.Journal of the American Planning Association. 1998;64 (4): 457-69.
30. Edwards Mary,Haines Anna.Evaluating Smart Growth: Implications for Small Communities.Journal of Planning Education and Research. 2007;27 (1): 49-64.
31. Edwards Ward,Newman J. RobertMultiattribute Evaluation. Thousand Oaks, CA: Sage; 1982:.
32. Evans-Cowley Jennifer S.,Gough Meghan Z.Evaluating Environmental Protection in Post-Hurricane Katrina Plans in Mississippi.Journal of Environmental Planning and Management. 2008;51 (3): 399-419.
33. Evans-Cowley Jennifer S.,Gough Meghan Z.Evaluating New Urbanist Plans in Post-Katrina Mississippi.Journal of Urban Design. 2009;14 (4): 439-61.
34. Evenson Kelly R.,Satinsky Sara B.,Rodriguez Daniel A.,Aytur Semra A.Exploring a Public Health Perspective on Pedestrian Planning.Health Promotion Practice. 2012;13 (2): 204-13.
35. Freelon Deen G.ReCal: Intercoder Reliability Calculation as a Web Service.International Journal of Internet Science. 2010;5 (1): 20-33.
36. Godschalk David R.,Beatley Timothy,Berke Philip,Brower David J.,Kaiser Edward J.Natural Hazard Mitigation: Recasting Disaster Policy and Planning. Washington, DC: Island Press; 1999:.
37. Hamin Elisabeth M.Climate Change and Land Policies. Ingram Gregory K.Hong Yu-Hung, ed. Cambridge: Lincoln Institute of Land Policy; 2011:.
38. Hayes Andrew F.,Krippendorff Klaus.Answering the Call for a Standard Reliability Measure for Coding Data.Communication Methods and Measures. 2007;1 (1): 77-89.
39. Hoch Charles.How Plan Mandates Work: Affordable Housing in Illinois.Journal of the American Planning Association. 2007;73 (1): 86-99.
40. Horney Jennifer A.,Naimi Ashley I.,Lyles Ward,Simon Matt,Salvesen David,Berke Philip.Assessing the Relationship between Hazard Mitigation Plan Quality and Rural Status in a Cohort of 57 Counties from 3 States in the Southeastern U.S.Challenges. 2012;3 (2): 183-93.
41. Jones David K.,Evenson Kelly R.,Rodriguez Daniel A.,Aytur Semra A.Addressing Pedestrian Safety: A Content Analysis of Pedestrian Master Plans in North Carolina.Traffic Injury Prevention. 2010;11 (1): 57-65.
42. Kaiser Edward J.,Davies John.What a Good Plan Should Contain: A Proposed Model.Carolina Planning. 1999;24 (2): 29-41.
43. Kang Jung Eun,Peacock Walter G.,Husein Rahmawati.An Assessment of Coastal Zone Hazard Mitigation Plans in Texas.Journal of Disaster Research. 2010;5 (5): 520-28.
44. Krippendorff Klaus.Reliability in Content Analysis: Some Common Misconceptions and Recommendations.Human Communication Research. 2004;30 (3): 411-33.
45. Krippendorff Klaus.Agreement and Information in the Reliability of Coding.Communication Methods and Measures. 2011;5 (2): 93-112.
46. Krippendorff KlausContent Analysis: An Introduction to Its Methodology. 2013. Content Analysis: An Introduction to Its Methodology, 3rd ed.Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2013:.
47. Lombard Matthew,Snyder-Duch Jennifer,Bracken Cheryl Campanella.Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability.Human Communication Research. 2002;28 (4): 587-604.
48. Manta-Conroy Maria,Berke Philip R.What Makes a Good Sustainable Development Plan? An Analysis of Factors That Influence Principles of Sustainable Development.Environment and Planning A. 2004;36 (8): 1381-96.
49. Miles Matthew B.,Huberman A. MichaelQualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, CA: Sage; 1984:.
50. Miles Matthew B.,Huberman A. MichaelQualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage; 1994:.
51. Nelson Arthur C.,French Steven P.Plan Quality and Mitigating Damage from Natural Disasters: A Case Study of the Northridge Earthquake with Planning Policy Considerations.Journal of the American Planning Association. 2002;68 (2): 194-207.
52. Norton Richard K.Local Commitment to State-Mandated Planning in Coastal North Carolina.Journal of Planning Education and Research. 2005a;25 (2): 149-71.
53. Norton Richard K.More and Better Local Planning: State-Mandated Local Planning in Coastal North Carolina.Journal of the American Planning Association. 2005b;71 (1): 55-72.
54. Norton Richard K.Using Content Analysis to Evaluate Local Master Plans and Zoning Codes.Land Use Policy. 2008;25 (3): 432-54.
55. Olonilua Oluponmile O.,Ibitayo Olurominiyi O.Toward Multihazard Mitigation: An Evaluation of FEMA-Approved Hazard Mitigation Plans under the Disaster Mitigation Act of 2000.Journal of Emergency Management. 2011;9 (1): 37-49.
56. Preston Benjamin L.,Westaway Richard M.,Yuen Emma J.Climate Adaptation Planning in Practice: An Evaluation of Adaptation Plans from Three Developed Nations.Mitigation and Adaptation Strategies for Global Change. 2011;16 (4): 407-38.
57. Putt Allen D.,Springer J.FredPolicy Research: Concepts, Methods, and Applications. Englewood Cliffs, NJ: Prentice Hall; 1989:.
58. Scott William A.Reliability of Content Analysis: The Case of Nominal Scale Coding.Public Opinion Quarterly. 1955;19 (3): 321-25.
59. Srivastava Rahul,Laurian Lucie.Natural Hazard Mitigation in Local Comprehensive Plans: The Case of Flood, Wildfire and Drought Planning in Arizona.Disaster Prevention and Management. 2006;15 (3): 461-83.
60. Stone Brian,Vargo Jason,Habeeb Dana.Managing Climate Change in Cities: Will Climate Action Plans Work?.Landscape and Urban Planning. 2012;107 (3): 263-71.
61. Talen Emily.Do Plans Get Implemented? A Review of Evaluation in Planning.Journal of Planning Literature. 1996;10 (3): 248-59.
62. Talen Emily,Knaap Gerrit.Legalizing Smart Growth: An Empirical Study of Land Use Regulation in Illinois.Journal of Planning Education and Research. 2003;22 (4): 345-59.
63. Tang Zhenghong.Evaluating Local Coastal Zone Land Use Planning Capacities in California.Ocean & Coastal Management. 2008;51 (7): 544-55.
64. Tang Zhenghong,Brody Samuel D.Linking Planning Theories with Factors Influencing Local Environmental-Plan Quality.Environment and Planning B. 2009;36 (6): 522-37.
65. Tang Zhenghong,Brody Samuel D.,Quinn Courtney,Chang Liang,Wei Ting.Moving from Agenda to Action: Evaluating Local Climate Change Action Plans.Journal of Environmental Planning and Management. 2010;53 (1): 41-62.
66. Tang Zhenghong,Lindell Michael K.,Prater Carla,Wei Ting,Hussey Christopher M.Examining Local Coastal Zone Management Capacity in U.S. Pacific Coastal Counties.Coastal Management. 2011;39 (2): 105-32.
67. Termorshuizen Jolande W.,Paul Opdam,van den Brink Adri.Incorporating Ecological Sustainability into Landscape Planning.Landscape and Urban Planning. 2007;79 (3-4): 374-84.
68. Wesman Alexander G.Educational Measurement. Thorndike Robert L., ed. Washington, DC: American Council in Education; 1971:.
69. Wheeler Stephen M.State and Municipal Climate Change Plans.Journal of the American Planning Association. 2008;74 (4): 481-96.
70. Wimmer Roger D.,Dominick Joseph R.Mass Media Research: An Introduction. Belmont, CA: Wadsworth; 1983:.