The Potential Application of Qualitative Evaluation Methods in European Regional Development: Reflections on the Use of Performance Story Reporting in Australian Natural Resource Management

Regional Studies

Volumn 49, Issue 8, 2015, Pages 1326-1339

  Vanclay, Frank ^a  

^a UNIVERSITY OF GRONINGEN   (Netherlands)

Author keywords

Impact evaluation; Narrative evaluation; Outcomes assessment; Qualitative evaluation; Retrospective evaluation; Story based approaches

Indexed keywords

EID: 84937515057     PISSN: 00343404     EISSN: 13600591     Source Type: Journal    
DOI: 10.1080/00343404.2013.837998     Document Type: Article

References (75)

1. J.Anatas (2004) Quality in qualitative evaluation: Issues and possible answers, Research on Social Work Practice 14(1), 57–65. doi: 10.1177/1049731503257870
2. Auditor General (2008) Regional Delivery Model for the Natural Heritage Trust and the National Action Plan for Salinity and Water Quality. Audit Report Number 21 (2007–08). Australian National Audit Office, Canberra, ACT (available at: http://www.anao.gov.au/uploads/documents/2007-08_Audit_Report_21.pdf) (accessed on 11 April 2012).
3. C.Bennett (1975) Up the hierarchy, Journal of Extension 13(2), 7–12.
4. Bessen Consulting Services (2009) Performance Story Report: Evaluation of Investment in the Dugong and Marine Turtle Project. North Australian Indigenous Land and Sea Management Alliance, Darwin, NT (available at: http://www.savanna.org.au/nailsma/publications/downloads/Dugong-and-Marine-Turtle-Project-PSR.pdf) (accessed on 28 April 2013).
5. A.Carr and F.Woodhams (2008) Monitoring, Evaluating, Reporting and Improving (MERI) Natural Resource Management: Making MERI in Australia. Bureau of Rural Sciences, Canberra, ACT.
6. D.Cash, W.Clark, F.Alcock, N.Dickson, N.Eckley, D.Guston, J.Jäger and R.Mitchell (2003) Knowledge systems for sustainable development, Proceedings of the National Academy of Sciences, USA 100(14), 8086–8091. doi: 10.1073/pnas.1231332100
7. Centers for Disease Control and Prevention (CDCP) (2007) Impact and Value: Telling Your Program’s Story. CDCP, National Centers for Chronic Disease Prevention and Health Promotion, Division of Oral Health, Atlanta, GA (available at: http://www.cdc.gov/oralhealth/publications/library/pdf/success_story_workbook.pdf) (accessed on 28 April 2013).
8. Clear Horizon and Environment Victoria (2010) Regional Sustainable Living Program: Performance Story Report. Environment Victoria, Melbourne, VIC (available at: http://environmentvictoria.org.au/sites/default/files/RSL%20Whole%20of%20Program%20Report%20-%20public%20version.pdf) (accessed on 28 April 2013).
9. Commonwealth of Australia (2008) Caring for Our Country Outcomes 2008–2013. Commonwealth of Australia, Canberra, ACT (available at: http://nrmonline.nrm.gov.au/catalog/mql:1887) (accessed on 28 April 2013).
10. Commonwealth of Australia (2012) One Land – Many Stories: Prospectus for Investment. Australian Government Department of Sustainability, Environment, Water, Population and Communities, Canberra, ACT (available at: http://www.environment.gov.au/biodiversity/publications/prospectus/index.html) (accessed on 25 July 2013).
11. A.Curtis and M.Lockwood (2000) Landcare and catchment management in Australia: Lessons for state-sponsored community participation, Society and Natural Resources 13(1), 61–73. doi: 10.1080/089419200279243
12. M.Dare, F.Vanclay and J.Schirmer (2011) Understanding community engagement in plantation forest management: Insights from practitioner and community narratives, Journal of Environmental Planning and Management 54(9), 1149–1168. doi: 10.1080/09640568.2011.560456
13. J.Dart and R.Davies (2003) A dialogical, story-based evaluation tool: the Most Significant Change Techniquef, American Journal of Evaluation 24(2), 137–155.
14. J.Dart, G.Drysdale, D.Cole and M.Saddington (2000) The Most Significant Change Approach for Monitoring an Australian Extension Project. PLA Notes Number 38, pp. 47–53. International Institute for Environment and Development, London.
15. J.Dart and J.Mayne (2005) Performance story, in S.Mathison (Ed.) Encyclopedia of Evaluation, pp. 307–308. Sage, Thousand Oaks, CA.
16. J.Dart and P.O'Connor (2008) Performance Story Report: A Study of the Mount Lofty Ranges Southern Emu-Wren and Fleurieu Peninsula Swamps Recovery Program and How it Contributed to Biodiversity Outcomes in the Adelaide and Mount Lofty Ranges Natural Resources Management Region. Commonwealth of Australia, Canberra, ACT (available at: http://nrmonline.nrm.gov.au/catalog/mql:2167) (accessed on 11 April 2012).
17. R.Davies and J.Dart (2005) The ‘Most Significant Change’ (MSC) Technique: A Guide to its Use [self-published manual for a consortium of development agencies] (available at: http://www.mande.co.uk/docs/MSCGuide.pdf) (accessed on 11 April 2012).
18. S.Denning (2007) The Secret Language of Leadership: How Leaders Inspire Action through Narrative. Jossey-Bass, San Francisco, CA.
19. Department of Water, Land and Biodiversity Conservation (DWLBC) (2009) Our Changing Environment: Outcomes and Lessons from Investment in South Australia’s Natural Resources 2001–2008, Government of South Australia, Department of Water, Land and Biodiversity Conservation, Adelaide, SA (available at: http://nrmonline.nrm.gov.au/catalog/mql:2677) (accessed on 10 October 2013).
20. G.Enticott and F.Vanclay (2011) The moral accountability of agricultural scripts: an analysis of farmers’ talk about animal health risks, Health Risk and Society 13(4), 293–309. doi: 10.1080/13698575.2011.575456
21. A.M.Esteves, D.Franks and F.Vanclay (2012) Social impact assessment: the state of the art, Impact Assessment and Project Appraisal 30(1), 35–44. doi: 10.1080/14615517.2012.660356
22. A.M.Esteves and F.Vanclay (2009) Social development needs analysis as a tool for SIA to guide corporate–community investment: applications in the minerals industry, Environmental Impact Assessment Review 29(2), 137–145. doi: 10.1016/j.eiar.2008.08.004
23. EVALSED (2012) EVALSED: The Resource for the Evaluation of Socioeconomic Development (available at: http://ec.europa.eu/regional_policy/sources/docgener/evaluation/guide/guide2012_evalsed.pdf) (accessed on 14 September 2013).
24. I.Feller (2007) Mapping the frontiers of evaluation of public sector R&D programs, Science and Public Policy 34(10), 681–690. doi: 10.3152/030234207X258996
25. D.M.Fetterman (2000) Foundations of Empowerment Evaluation. Sage, Thousand Oaks, CA.
26. D.M.Fetterman, S.J.Kaftarian and A.Wandersman (Eds) (1995) Empowerment Evaluation. Sage, Thousand Oaks, CA.
27. W.R.Fisher (1989) Human Communication as Narration: Toward a Philosophy of Reason, Value, and Action. University of South Carolina Press, Columbia, SC.
28. J.C.Greene (2000) Qualitative program evaluation: practice and promise, in N.K.Denzin and Y.S.Lincoln (Eds) Handbook of Qualitative Research, pp. 530–544. Sage, Thousand Oaks, CA.
29. E.Guba and Y.Lincoln (1989) Fourth Generation Evaluation. Sage, Newbury Park, CA.
30. I.Guijt, J.Brouwers, C.Kusters, E.Prins and B.Zeynalova (2011) Evaluation Revisited: Improving the Quality of Evaluative Practice by Embracing Complexity (Conference Report). Centre for Development Innovation, Wageningen University and Research Centre, Wageningen (available at: http://capacity.org/capacity/export/sites/capacity/documents/topic-readings/110412-evaluation-revisited-may-2010_small-version.pdf) (accessed on 11 April 2012).
31. S.Hajkowicz (2009) The evolution of Australia's natural resource management programs: towards improved targeting and evaluation of investments, Land Use Policy 26(2), 471–478. doi: 10.1016/j.landusepol.2008.06.004
32. P.Howden and F.Vanclay (2000) Mythologization of farming styles in Australian broadacre cropping, Rural Sociology 65(2), 295–310. doi: 10.1111/j.1549-0831.2000.tb00030.x
33. R.Hughes and M.Huby (2002) The application of vignettes in social and nursing research, Journal of Advanced Nursing 37(4), 382–386. doi: 10.1046/j.1365-2648.2002.02100.x
34. W.Hunt, F.Vanclay, C.Birch, J.Coutts, J.Flittner and B.Williams (2011) Agricultural extension: Building capacity and resilience in rural industries and communities, Rural Society 20(2), 112–127. doi: 10.5172/rsj.20.2.112
35. E.T.Jackson and Y.Kassam (Eds) (1998) Knowledge Shared: Participatory Evaluation in Development Cooperation. Kumarian, West Hartford, CT.
36. E.João, F.Vanclay and L.den Broeder (2011) Emphasising enhancement in all forms of impact assessment, Impact Assessment and Project Appraisal 29(3), 170–180. doi: 10.3152/146155111X12959673796326
37. C.Kurtz (2009) Working with Stories in your Community or Organization (available at: http://www.workingwithstories.org) (accessed on 11 April 2012).
38. C.Kurtz and D.Snowden (2003) The new dynamics of strategy: sense-making in a complex and complicated world, IBM Systems Journal 42(3), 462–483. doi: 10.1147/sj.423.0462
39. R.Labonte (2011) Reflections on stories and a story/dialogue method in health research, International Journal of Social Research Methodology 14(2), 153–163. doi: 10.1080/13645579.2010.492131
40. R.Labonte, J.Feather and M.Hills (1999) A story/dialogue method for health promotion knowledge development and evaluation, Health Education Research 14(1), 39–50. doi: 10.1093/her/14.1.39
41. R.Laughlin and J.Broadbent (1996) Redesigning fourth generation evaluation, Evaluation 2(4), 431–451. doi: 10.1177/135638909600200406
42. R.Lavinghouze., A.Price and K.Smith (2007) The program success story: a valuable tool for program evaluation, Health Promotion Practice 8(4), 323–331. doi: 10.1177/1524839907306408
43. F.Leeuw and J.Vaessen (2009) Impact Evaluations and Development: NONIE Guidance on Impact Evaluation. The Network of Networks on Impact Evaluation. c/- Independent Evaluation Group, The World Bank, Washington, DC (available at: http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf) (accessed on 11 April 2012).
44. M.Maxson, I.Guijt, (2010) The ‘Real Book’ for Story Evaluation Methods. GlobalGiving Foundation (available at: http://www.globalgiving.org/jcr-content/gg/landing-pages/story-tools/files/-story-real-book--2010.pdf) (accessed on 11 April 2012).
45. J.Mayne (1999) Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly (Discussion Paper). Office of Auditor-General of Canada (available at: http://dsp-psd.pwgsc.gc.ca/Collection/FA3-31-1999E.pdf) (accessed on 11 April 2012).
46. J.Mayne (2004) Reporting on outcomes: setting performance expectations and telling performance stories, Canadian Journal of Program Evaluation 19(1), 31–60.
47. D.K.Padgett (2012) Qualitative and Mixed Methods in Public Health. Sage, Thousand Oaks, CA.
48. M.Q.Patton (1981) Creative Evaluation. Sage, Beverly Hills, CA.
49. M.Q.Patton (1990) Qualitative Evaluation and Research Methods. Sage, Newbury Park, CA.
50. M.Q.Patton (1997) Utilization-Focused Evaluation. Sage, Thousand Oaks, CA.
51. K.Prager and F.Vanclay (2010) Landcare in Australia and Germany: comparing structures and policies for community engagement in natural resource management, Ecological Management and Restoration 11(3), 187–193. doi: 10.1111/j.1442-8903.2010.00548.x
52. L.Robins and P.Kanowski (2011) ‘Crying for our Country’: eight ways in which ‘Caring for our Country’ has undermined Australia's regional model for natural resource management, Australasian Journal of Environmental Management 18(2), 88–108. doi: 10.1080/14486563.2011.566158
53. N.Roese (1997) Counterfactual thinking, Psychological Bulletin 121(1), 133–148. doi: 10.1037/0033-2909.121.1.133
54. A.Roughley (2009) Developing and Using Program Logic in Natural Resource Management: User Guide. Australian Government, Canberra, ACT (available at: http://nrmonline.nrm.gov.au/catalog/mql:2164) (accessed on 11 April 2012).
55. A.Roughley and J.Dart (2009) Developing a Performance Story Report. User Guide. Australian Government, Canberra, ACT (available at: http://nrmonline.nrm.gov.au/catalog/mql:2162) (accessed on 11 April 2012).
56. D.Royce, B.T.Thyer, D.K.Padgett and T.K.Logan (2001) Program Evaluation: An Introduction, 3rd Edn. Brooks/Cole, Pacific Grove, CA.
57. M.Sandelowski (1991) Telling stories: narrative approaches in qualitative research, Image: Journal of Nursing Scholarship 23(3), 161–166.
58. G.Shaw, R.Brown and P.Bromiley (1998) Strategic stories: how 3M is rewriting business planning, Harvard Business Review 76(3), 41–50.
59. M.Silver, R.Richards and S.Cowell (2009) A Guide to Designing and Implementing Performance Reporting to Increase the Confidence of Conservation Investors. Bush Heritage Australia, Melbourne, VIC (available at: http://lwa.gov.au/files/products/reporting/pn30209/pn30209.pdf) (accessed on 11 April 2012).
60. D.Snowden and M.Boone (2007) A leader's framework for decision making, Harvard Business Review 85(11), 69–76.
61. R.Tennent and S.Lockie (2013) Vale Landcare: the rise and decline of community-based natural resource management in rural Australia, Journal of Environmental Planning and Management 56(4), 572–587. doi: 10.1080/09640568.2012.689617
62. F.Vanclay (2003) International principles for social impact assessment, Impact Assessment and Project Appraisal 21(1), 5–11. doi: 10.3152/147154603781766491
63. F.Vanclay (2004) Social principles for agricultural extension to assist in the promotion of natural resource management, Australian Journal of Experimental Agriculture 44(3), 213–222. doi: 10.1071/EA02139
64. F.Vanclay (2012a) Guidance for the Design of Qualitative Case Study Evaluation: A Short Report to DG Regio (available at: http://ec.europa.eu/regional_policy/sources/docgener/evaluation/doc/performance/Vanclay.pdf) (accessed on 17 May 2012).
65. F.Vanclay (2012b) The potential application of Social Impact Assessment in coastal zone management, Ocean and Coastal Management 68, 149–156. doi: 10.1016/j.ocecoaman.2012.05.016
66. F.Vanclay and G.Enticott (2011) The role and functioning of cultural scripts in farming and agriculture, Sociologia Ruralis 51(3), 256–271. doi: 10.1111/j.1467-9523.2011.00537.x
67. F.Vanclay, R.Lane, J.Wills, I.Coates and D.Lucas (2004) ‘Committing to Place’ and evaluating the higher purpose: Increasing engagement in natural resource management through museum outreach and educational activities, Journal of Environmental Assessment Policy and Management 6(4), 539–564. doi: 10.1142/S1464333204001791
68. Y.Wadsworth (2005) ‘Gouldner's child?’: some reflections on sociology and participatory action research, Journal of Sociology 41(3), 267–284. doi: 10.1177/1440783305057079
69. Y.Wadsworth, A.Wierenga and G.Wilson (2007) Writing Narrative Action Evaluation Reports in Health Promotion: Manual of Guidelines, Resources, Case Studies and QuickGuide, 2nd Edn. State of Victoria, Department of Human Services and the University of Melbourne, Melbourne, VIC.
70. D.Waters, R.James and J.Darby (2011) Health promoting community radio in rural Bali: An impact evaluation, Rural and Remote Health 11(1), article #1555 (online journal available at: http://www.rrh.org.au/articles/subviewnew.asp?ArticleID=1555) (accessed on 11 April 2012).
71. E.Whitmore (Ed.) (1998) Understanding and Practicing Participatory Evaluation: New Directions for Evaluation. Jossey-Bass, San Francisco, CA.
72. W.F.Whyte (Ed.) (1990) Participatory Action Research. Sage, Newbury Park, CA.
73. L.Wilder and M.Walpole (2008) Measuring social impacts in conservation: experience of using the Most Significant Change method, Oryx 42(4), 529–538. doi: 10.1017/S0030605307000671
74. J.Willetts and P.Crawford (2007) The most significant lessons about the Most Significant Change Technique, Development in Practice 17(3), 367–379. doi: 10.1080/09614520701336907
75. D.Withers (2010) Narrative Evaluation: City of Surrey's Cultural Capital of Canada Award Program (available at: http://denisewithers.files.wordpress.com/2011/02/surreycccreport1.pdf) (accessed on 28 April 2013).