Unlocking the Potential of the “What Works” Approach to Policymaking and Practice: Improving Impact Evaluations

American Journal of Evaluation

Volumn 36, Issue 4, 2015, Pages 558-569

  Granger, Robert C ^a   Maynard, Rebecca ^b  

^a Independent Consultant   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

evaluation policy; evaluation practice; evaluation use; impact evaluation

Indexed keywords

EID: 84947442409     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214015594420     Document Type: Article

References (50)

1. Abt Associates. (2014). Learning from the National Evaluation of i3: Challenges, responses, and future plans i3 Learning Community. Bethesda, MD. Presentation for the I-3 Learning Community Meeting, September 11–12, 2014. Retrieved fromhttp://forumfyi.org/files/Learning_National_Eval_i3.pdf
2. J.AnhøjC.Møldrup (2004). Feasibility of collecting diary data from asthma patients through mobile phones and SMS (short message service): Response rate analysis and focus group evaluation from a pilot study. Journal of Medical Internet Research, 6, e42.
3. C.G.BegleyJ.P.Ioannidis (2015). Reproducibility in science improving the standard for basic and preclinical research. Circulation Research, 116, 116–126.
4. K.BollerD.DaroP.D.GrossoR.P.ColeD.PaulsellB.Hart…M.Hargreaves (2014). Making replication work: Building infrastructure to implement, scale-up, and sustain evidence-based early childhood home visiting programs with fidelity. Washington, DC: Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services. Retrieved fromhttp://www.chapinhall.org/sites/default/files/documents/EBHV_MakingReplication_Final.pdf
5. L.M.BranscombJ.H.Keller (1999). Investing in innovation: Creating a research and innovation policy that works. Cambridge, MA: MIT Press.
6. Coalition for Evidence-Based Policy. (2013). Randomized controlled trials commissioned by the Institute of Education Sciences since 2002: How many found positive versus weak or no effects. Washington, DC: Author. Retrieved fromhttp://coalition4evidence.org/wp-content/uploads/2013/06/IES-Commissioned-RCTs-positive-vs-weak-or-null-findings-7-2013.pdf
7. J.Cohen (2013). Statistical power analysis for the behavioral sciences. San Diego, CA: Academic Press.
8. K.M.CresswellS.SadlerS.RodgersA.AveryJ.CantrillS.A.MurrayA.Sheikh (2012). An embedded longitudinal multi-faceted qualitative evaluation of a complex cluster randomized controlled trial aiming to reduce clinically important errors in medicines management in general practice. Trials, 13, 78.
9. N.DongR.Maynard (2013). PowerUP!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6, 24–67.
10. G.J.DuncanM.EngelA.ClaessensC.J.Dowsett (2014). Replication and robustness in developmental research. Developmental Psychology, 50, 2417.
11. J.A.DurlakE.P.DuPre (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.
12. J.EarleR.MaynardR.Neild (2013). Common evidence guidelines for education research. Washington, DC: Institute of Education Sciences and National Science Foundation. Retrieved fromhttp://ies.ed.gov/pdf/CommonGuidelines.pdf
13. P.J.EasterbrookR.GopalanJ.A.BerlinD.R.Matthews (1991). Publication bias in clinical research. The Lancet, 337, 867–872.
14. D.Fanelli (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.
15. C.J.FergusonM.T.Brannick (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120.
16. D.L.FixsenS.F.NaoomK.A.BlaseR.M.FriedmanF.Wallace (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Retrieved fromhttp://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-MonographFull-01-2005.pdf
17. J.GueronH.Rolston (2013). Fighting for reliable evidence (p. 596). New York, NY: Russell Sage Foundation.
18. R.HaskinsJ.Baron (2011). Building the connection between policy and evidence. London, England: NESTA. Retrieved fromhttp://coalition4evidence.org/wp-content/uploads/uploads-dupes-safety/Haskins-Baron-paper-on-fed-evid-based-initiatives-2011.pdf
19. R.HaskinsG.Margolis (2014). Show me the evidence: Obama’s fight for rigor and results in social policy. Washington, DC: Brookings Institution Press.
20. C.S.HullemanD.S.Cordray (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.
21. K.ImaiL.KeeleD.TingleyT.Yamamoto (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105, 765–789.
22. Institute of Education Sciences. (n.d.). National Center for Education Statistics. Retrieved fromhttp://nces.ed.gov/programs/slds/
23. M.KelseyA.JohnsonR.Maynard (2001). The potential of home visitor services to strengthen welfare-to-work programs for teenage parents on cash assistance. Philadelphia: University of Pennsylvania and Mathematica Policy Research.
24. S.T.Kew (2010). Text messaging: An innovative method of data collection in medical research. BMC Research Notes, 3, 342.
25. N.A.LandenbergerM.W.Lipsey (2005). The positive effects of cognitive-behavioral programs for offenders: A meta-analysis of factors associated with effective treatment. Journal of Experimental Criminology, 1, 451–476.
26. M.W.Lipsey (1993). Theory as method: Small theories of treatments. New Directions for Program Evaluation, 57, 5–38.
27. M.W.Lipsey (2009). Identifying interesting variables and analysis opportunities. In H.CooperL.V.HedgesJ.C.Valentine (Eds.). The handbook of research synthesis and meta-analysis (2nd ed., pp. 147–158). New York, NY: Russell Sage Foundation.
28. M.C.MakelJ.A.PluckerB.Hegarty (2012). Replications in psychology research: How often do they really occur?Perspectives on Psychological Science, 7, 537–542.
29. R.A.Maynard (2006). Presidential address: Evidence-based decision making: What will it take for the decision makers to care?Journal of Policy Analysis & Management, 25, 249–265.
30. S.McNeeleyJ.J.Warner (2015). Replication in criminology: A necessary practice. European Journal of Criminology. Advance online publication. doi:10.1177/1477370815578197
31. National Board for Education Sciences. (2008). National Board for Education Sciences: 5-Year Report, 2003 Through 2008(NBES 2009-6011). Washington, DC: Author. Retrieved fromhttp://ies.ed.gov/director/board/index.asp
32. M.C.NelsonD.S.CordrayC.S.HullemanC.L.DarrowE.C.Sommer (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. The Journal of Behavioral Health Services and Research, 39, 374–396.
33. A.N.NeuhoffS.AxworthyS.GlazerD.Berfond (2015). The what works marketplace: Helping leaders use evidence to make smarter choices. Boston, MA: The Bridgespan Group. Retrieved fromhttp://results4america.org/wp-content/uploads/2015/04/WhatWorksMarketplace-vF.pdf
34. J.NussleP.Orszag (Eds.). (2014). Moneyball for Government. Washington, DC: Disruption Books.
35. M.E.O’ConnellT.BoatK.E.Warner (Eds.). (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press. Retrieved fromhttp://www.nap.edu/catalog.php?record_id=12480
36. L.L.Orr (1999). Social experiments: Evaluating public programs with experimental methods. A systematic approach. Thousand Oaks, CA: Sage.
37. S.PatnaikE.BrunskillW.Thies (2009). Evaluating the accuracy of data collection on mobile phones: A study of forms, SMS, and voice. Proceedings of the International Conference on Information and Communication Technologies and Development (pp. 74–84). doi:10.1109/ICTD.2009.5426700. Retrieved fromhttp://hdl.handle.net/1721.1/60077
38. T.D.PigottJ.C.ValentineJ.R.PolaninR.T.Williams (2013). Outcome-reporting bias in education research. Educational Researcher, 2, 424–432.
39. S.W.RaudenbushA.MartinezJ.Spybrook (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29, 5–29.
40. P.H.RossiM.W.LipseyH.E.Freeman (2003). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.
41. H.R.RothsteinA.J.SuttonM.Borenstein (Eds.). (2006). Publication bias in meta-analysis: Prevention, assessment and adjustments. Hoboken, NJ: John Wiley & Sons.
42. J.SpybrookH.BloomR.CongdonC.HillA.MartinezS.Raudenbush (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software. New York, NY: William T. Grant Foundation.
43. S.Tanner (2015). Evidence of false positives in research clearinghouses and influential journals: An application of P-curve to policy research. Observational Studies, 1, 18–29.
44. M.TomlinsonW.SolomonY.SinghT.DohertyM.ChopraP.IjumbaD.Jackson (2009). The use of mobile phones as a data collection tool: A report from a household survey in South Africa. BMC Medical Informatics and Decision Making, 12, 236. Retrieved fromhttp://www.ncbi.nlm.nih.gov/pmc/articles/PMC3248873/pdf/1745-6215-12-236.pdf
45. U.S. Department of Education. (2013). What works clearinghouse: Procedure and standards handbook(Version 3.0). Washington, DC: National Center for Education Evaluation and Regional Assistance. Retrieved fromhttp://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_standards_handbook.pdf
46. U.S. Department of Education. (2014). Request for applications for education research grants CFDA Number: 84.305A. Washington, DC: Institute of Education Sciences (pp. 62–66). Retrieved fromhttp://ies.ed.gov/funding/pdf/2015_84305A.pdf
47. U.S. Department of Justice. (2014). Model program guide. Washington, DC: Office of Juvenile Justice and Delinquency Prevention. Retrieved fromhttp://www.ojjdp.gov/mpg/Home/About#HowTo
48. W. K. Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI: W. K. Kellogg Foundation. Retrieved fromhttp://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
49. M.WeissH.BloomT.Brock (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33, 778–808.
50. D.B.WilsonM.W.Lipsey (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6, 413.