Design of an impact evaluation using a mixed methods model - An explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi

BMC Health Services Research

Volumn 14, Issue 1, 2014, Pages

  Brenner, Stephan ^a   Muula, Adamson S ^b   Robyn, Paul Jacob ^c   Bärnighausen, Till ^d,e   Sarker, Malabika ^a   Mathanga, Don P ^b   Bossert, Thomas ^d   De Allegri, Manuela ^a  

^a UNIVERSITY OF HEIDELBERG   (Germany)

^b UNIVERSITY OF MALAWI   (Malawi)

^c WORLD BANK   (United States)

^d HARVARD SCHOOL OF PUBLIC HEALTH   (United States)

^e UNIVERSITY OF KWAZULU NATAL   (South Africa)

Author keywords

Impact evaluation; Mixed methods; Performance based incentives; Study design

Indexed keywords

ECONOMICS; EVALUATION STUDY; FEMALE; HEALTH SERVICE; HUMAN; MALAWI; REIMBURSEMENT;

EVALUATION STUDIES AS TOPIC; FEMALE; HUMANS; MALAWI; MATERNAL HEALTH SERVICES; REIMBURSEMENT, INCENTIVE;

EID: 84899987684     PISSN: None     EISSN: 14726963     Source Type: Journal    
DOI: 10.1186/1472-6963-14-180     Document Type: Article

References (73)

1. Towards universal health care coverage: a goal-oriented framework for policy analysis. Kutzin J, Health, nutrition and population discussion paper Washington D.C: The World Bank, Preker AS, 2000
2. Gottret PE, Schieber G, Health financing revisited: a practitioner's guide Washington, D.C: The World Bank 2006
3. World Health Assembly Resolution 58.33, Sustainable health financing, universal coverage and social health insurance Geneva: World Health Organization 2005
4. Advancement of global health: key messages from the disease control priorities project. Laxminarayan R, Mills AJ, Breman JG, Measham AR, Alleyne G, Claeson M, Jha P, Musgrove P, Chow J, Shahid-Salles S, Jamison DT, Lancet 2006 367 1193 1208 10.1016/S0140-6736(06)68440-7 16616562
5. World Health Organization, The world health report 2010-health systems financing: the path to universal coverage Geneva: World Health Organization 2010
6. Missing in action: teacher and health worker absence in developing countries. Chaudhury N, Hammer J, Kremer M, Muralidharan K, Rogers FH, J Econ Perspect 2006 20 91 116 10.1257/089533006776526058 17162836
7. Measuring client satisfaction and the quality of family planning services: a comparative analysis of public and private health facilities in Tanzania, Kenya and Ghana. Hutchinson PL, Do M, Agha S, BMC Health Serv Res 2011 11 203 219 10.1186/1472-6963-11-203 21864335
8. PETS-QSDS in sub-Saharan Africa: a stocktaking study. Gauthier B, Report for the project measuring progress in public services delivery: 7 September 2006 Washington D.C: The World Bank 2006
9. Public spending on health care and the poor. Gupta S, Verhoeven M, Tiongson ER, Health Econ 2003 12 685 696 10.1002/hec.759 12898665
10. Financial and other rewards for good performance or results: a guided tour of concepts and terms and a short glossary. Musgrove P, The World Bank health results innovation trust fund-results-based financing for health: 14 September 2010 Washington D.C: The World Bank 2011
11. Toonen J, van der Wal B, Results-based financing in healthcare: developing an RBF approach for healthcare in different contexts: the case of Mali and Ghana Amsterdam: KIT Publishers 2012
12. Creating stronger incentives for high-quality health care in low- and middle-income countries. Ergo A, Paina L, Morgan L, Eichler R, USAID maternal and child health integrated program Washington D.C: USAID 2012
13. Eichler R, Levine R, Performance-Based Incentives Working Group, Performance incentives for global health-potential and pitfalls Washington D.C: Center for Global Development 2009
14. Can performance-based financing be used to reform health systems in developing countries. Ireland M, Paul E, Dujardin B, Bull World Health Organ 2011 89 695 698 10.2471/BLT.11.87379 21897491
15. Performance-based financing: just a donor fad or a catalyst towards comprehensive health-care reform? Meessen B, Soucat A, Sekabaraga C, Bull World Health Organ 2011 89 153 156 10.2471/BLT.10.077339 21346927
16. Promoting healthy behaviours and improving health outcomes in low and middle income coutnries: a review of the impact of conditional cash transfer programmes. Ranganathan M, Legarde M, Prev Med 2012 55 95 S105 22178043
17. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. Giuffrida A, Gravelle H, Roland M, BMJ 1999 319 94 98 10.1136/bmj.319.7202.94 10398635
18. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Basinga P, Gertler PJ, Binagwaho A, Soucat AL, Sturdy J, Vermeersch CM, Lancet 2011 377 1421 1428 10.1016/S0140-6736(11)60177-3 21515164
19. Performance-based financing for better quality of services in Rwandan health centres: 3-year experience. Rusa L, Ngirabega J de D, Janssen W, Van Bastelaere S, Porignon D, Vandenbulcke W, Trop Med Int Health 2009 14 830 837 10.1111/j.1365-3156.2009.02292.x 19497081
20. Performance-based financing experiment improved health care in the democratic republic of Congo. Soeters R, Peerenboom PB, Mushagalusa P, Kimanuka C, Health Aff 2011 30 1518 1527 10.1377/hlthaff.2009.0019
21. Evidence review: results-based financing of maternal and newborn health care in low- and middle-income countries. Gorter A, Ir P, Meessen B, Report for the project Programme to Foster Innovation, Learning and Evidence in Health Programmes of the German Development Cooperation: study commissioned and funded by the German federal ministry for economic cooperation and development (BMZ) through the sector project PROFILE at GIZ-Deutsche Gesellschaft für Internationale Zusammenarbeit Eschborn: German Health Practice Collection 2013
22. Client satisfaction and perceived quality of primary health care in Uganda. Lundberg M, Are you being served? new tools for measuring service delivery Washington D.C: The World Bank, Amin S, Dasb J, Goldstein M, 2008 313 341
23. 'Paying for performance' in Rwanda: does it pay off? Kalk A, Paul FA, Grabosch E, Trop Med Int Health 2010 15 182 190 10.1111/j.1365-3156.2009.02430.x 19930141
24. Assessing performance enhancing tools: experiences with the open performance review and appraisal system (OPRAS) and expectations towards payment for performance (P4P) in the public health sector in Tanzania. Songstad NG, Lindkvist I, Moland KM, Chimhutu V, Blystad A, Global Health 2012 8 33 45 10.1186/1744-8603-8-33 22963317
25. Professionalism and the know-do gap: exploring intrinsic motivation among health workers in Tanzania. Leonard KL, Masatu MC, Health Econ 2010 19 1461 1477 10.1002/hec.1564 19960481
26. Paying for performance to improve the delivery of health interventions in low- and middle-income countries. Witter S, Fretheim A, Kessy FL, Lindahl AK, Cochrane Database Syst Rev 2012 2 1 81
27. Performance-based financing in low- and middle-income countries: still more questions than answers. Fretheim A, Witter S, Lindahl AK, Olsen IT, Bull World Health Organ 2012 90 559 559A 10.2471/BLT.12.106468 22893735
28. National Statistical Office and ICF Macro, Malawi demographic and health survey 2010 Zomba: National Statistical Office 2011
29. Facility-based maternal death review in three districts in the central region of Malawi: an analysis of causes and characteristics of maternal deaths. Kongnyuy EJ, Mlava G, van den Broek N, Womens Health Issues 2009 19 14 20 10.1016/j.whi.2008.09.008 19111783
30. Malawi health SWAp mid-term review: summary report. Carlson C, Boivin M, Chirwa A, Chirwa S, Chitalu F, Hoare G, Huelsmann M, Ilunga W, Maleta K, Marsden A, Martineau T, Minett C, Mlambala A, von Massow F, Njie H, Olson IT, Norad collected reviews, vol 22 Malawi: Norad: Commissioned by the Ministry of Health 2008
31. Assessing the use of an essential health package in a sector wide approach in Malawi. Bowie C, Mwase T, Health Research Policy and Systems 2011 9 4 13 10.1186/1478-4505-9-4 21241477
32. RBF4MNH Options Office, Inception report results based financing for maternal and neonatal health (RBF4MNH initiative) 2012-2014 Lilongwe: Reproductive Health Unit 2012
33. World Health Organization, Monitoring emergency obstetric care: a handbook Geneva: World Health Organization 2009
34. Maternal mortality: who, when, where, and why. Ronsmans C, Graham WJ, Lancet 2006 368 1189 1200 10.1016/S0140-6736(06)69380-X 17011946
35. Maternal health in poor countries: the broader context and a call for action. Filippi V, Ronsmans C, Campbell OM, Graham WJ, Mills A, Borghi J, Koblinsky M, Osrin D, Lancet 2006 368 1535 1541 10.1016/S0140-6736(06)69384-7 17071287
36. World Health Organization, Everybody's business: strengthening health systems to improve health outcomes: WHO's framework for action Geneva: World Health Organization 2007
37. The quality of care: how can it be assessed? Donabedian A, JAMA 1988 260 1743 1748 10.1001/jama.1988.03410120089033 3045356
38. Antenatal care services in rural Uganda missed opportunities for good-quality care. Conrad P, De Allegri M, Moses A, Larsson EC, Neuhann F, Müller O, Sarker M, Qual Health Res 2012 22 619 629 10.1177/ 1049732311431897 22232296
39. Revisiting the behavioral model and access to medical care: does it matter? Andersen RM, J Health Soc Behav 1995 36 1 10 10.2307/2137284 7738325
40. Equity of access to medical care: a conceptual and empirical overview. Aday LA, Andersen RM, Med Decis Making 1981 19 4 27
41. Creswell JW, Clark PVL, Designing and conducting mixed methods research Thousand Oaks: SAGE Publications 2 2010
42. Shadish WR, Cook TD, Campbell DT, Experimental and quasi-experimental designs for generalized causal inference Belmont: Wadsworth Cengage Learning 2001
43. Glaser BG, Strauss AL, The discovery of grounded theory: strategies for qualitative research Hawthorne: Aldine de Gruyter 1999
44. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. Peabody JW, JAMA 2000 283 1715 1722 10.1001/jama.283.13.1715 10755498
45. Quality of antenatal care in rural southern Tanzania: a reality check. Sarker M, Schmid G, Larsson E, Kirenga S, De Allegri M, Neuhann F, Mbunda T, Lekule I, Müller O, BMC Res Notes 2010 2010 3 209 315
46. A methodology for conducting retrospective chart review research in child and adolescent psychiatry. Gearing RE, Mian IA, Barber J, Ickowicz A, J Can Acad Child Adolesc Psychiatry 2006 15 126 134 18392182
47. Analyzing health equity using household survey data: a guide to techniques and their implementation. O'Donnell O, Van Doorslaer E, Wagstaff A, Lindelow M, WBI learning resource series Washington, D.C: The World Bank 2008
48. Inference with difference-in-differences and other panel data. Donald SG, Lang K, Rev Econ Stat 2007 89 221 233 10.1162/rest.89.2.221
49. Identification and inference in nonlinear difference-in-difference models. Athey S, Imbens GW, Econometrica 2006 74 431 497 10.1111/j.1468-0262. 2006.00668.x
50. Measuring impact in the millennium development goal era and beyond: a new approach to large-scale effectiveness evaluations. Victora CG, Black RE, Boerma JT, Bryce J, Lancet 2011 377 85 95 10.1016/S0140-6736(10)60810-0 20619886
51. The renaissance of field experimentation in evaluating interventions. Shadish WR, Cook TD, Annu Rev Psychol 2009 60 607 629 10.1146/annurev.psych.60. 110707.163544 18616391
52. The estimation of causal effects from observational data. Winship C, Morgan SL, Annu Rev Sociol 1999 25 659 707 10.1146/annurev.soc.25.1.659
53. Patton MQ, Qualitative research & evaluation methods Thousand Oaks: SAGE Publications 2002
54. Naturalistic inquiry and the saturation concept: a research note. Bowen GA, Qual Res 2008 8 137 152 10.1177/1468794107085301
55. Qualitative research sampling: the very real complexities. Tuckett AG, Nurse Researcher 2004 12 47 61 10.7748/nr2004.07.12.1.47.c5930 15493214
56. Opening the black box: using process evaluation measures to assess implementation and theory building. Harachi TW, Abbott RD, Catalano RF, Haggerty KP, Fleming CB, Am J Community Psychol 1999 27 711 731 10.1023/A:1022194005511 10676545
57. Process evaluation in randomised controlled trials of complex interventions. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, BMJ 2006 332 413 416 10.1136/bmj.332.7538.413 16484270
58. QSR International Pty Ltd, NVivo qualitative data analysis software 2012 Version 10
59. The evaluation of international development programs: a view from the front. Bamberger M, Am J Eval 2000 21 95 102 10.1177/109821400002100108
60. Exploring the potential for using results-based financing to address non-communicable diseases in low- and middle-income countries. Beane CR, Hobbs SH, Thirumurthy H, BMC Public Health 2013 13 92 100 10.1186/1471-2458-13-92 23368959
61. Funding aid according to outcomes can improve health in poor countries, seminar hears. Hurley R, BMJ 2011 342 2322 10.1136/bmj.d2322
62. Introduction to mixed methods in impact evaluation. Bamberger M, Impact evaluation guidance notes Washington D.C: InterAction 2012
63. Theory-based impact evaluation: principles and practice. White H, 3ie working papers New Delhi: International Initiative for Impact Evaluation (3ie) 2009
64. The theory-driven approach to validity. Chen H-T, Rossi PH, Evaluation and Program Planning 1987 10 95 103 10.1016/0149-7189(87)90025-5
65. World Health Organization, Country cooperation strategy brief Malawi Geneva: World Health Organization 2013
66. Motivation and incentives of rural maternal and neonatal health care providers: a comparison of qualitative findings from Burkina Faso, Ghana and Tanzania. Prytherch H, Kagone M, Aninanya GA, Williams JE, Kakoko DC, Leshabari MT, Ye M, Marx M, Sauerborn R, BMC Health Serv Res 2013 13 149 163 10.1186/1472-6963-13-149 23617375
67. Why performance-based contracting failed in Uganda: an 'open-box' evaluation of a complex health system intervention. Ssengooba F, McPake B, Palmer N, Soc Sci Med 2012 75 377 383 10.1016/j.socscimed.2012.02.050 22560799
68. Observatory AHW, Human resources for health country profile-Malawi Brazzaville: Republic of Congo 2009
69. Applying an equity lens to child health and mortality: more of the same is not enough. Victora CG, Wagstaff A, Schellenberg JA, Gwatkin D, Claeson M, Habicht JP, Lancet 2003 362 233 241 10.1016/S0140-6736(03)13917-7 12885488
70. Population Unit, Ministry of Development Planning and Cooperation, RAPID: population and development in Malawi Lilongwe: Ministry of Development Planning and Cooperation 2010
71. A contribution to current debates in impact. White H, Evaluation 2010 16 153 164 10.1177/1356389010361562
72. Morgan SL, Counterfactuals and causal inference: methods and principles for social research New York: Cambridge University Press 2007
73. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. Puffer S, Torgerson D, Watson J, BMJ 2003 327 785 789 10.1136/bmj.327.7418.785 14525877 Development