Looking inside the "Black Box": A methodology for measuring program implementation and informing social services policy decisions

Social Policy Journal

Volumn 4, Issue 3-4, 2006, Pages 1-17

  Carrilio, Terry ^a  

^a SAN DIEGO STATE UNIVERSITY   (United States)

Author keywords

Evidence based practice; Model fidelity; Outcomes; Program evaluation; Program implementation

Indexed keywords

EID: 33746570597     PISSN: 15332942     EISSN: 15332950     Source Type: Journal    
DOI: 10.1300/J185v04n03_01     Document Type: Article

References (66)

1. Bugental, D. B., Ellerson, P. C., Lin, E. K., Rainey, B., Kokotovic, A., & O'Hara, N. (2002). A cognitive approach to child abuse prevention. Journal of Family Psychology, 16(3), 243-258.
2. Buka, S., & Earls, F. (1993). Early determinants of delinquency and violence. Health Affairs, 12(4),46-64.
3. Burke, A. C., & Clapp, J. D. (1997). Ideology and social work practice in substance abuse settings. Social Work, 42(6), 552-562.
4. Carrilio, T. (1998). California Safe and Healthy Families: Executive Summary (2nd Ed.). Sacramento: California Department of Social Services.
5. Carrilio, T. (2001). Family support program development-integrating research, practice, and policy. Journal of Family Social Work, 6(3), 53-78.
6. Carrilio, T., Packard, T., Cannady, L., Darling, M., Chamberlin, J., Gamble, L., Bergthold, T., & Dorosky, E. (2002). Final report - Answers Benefitting Children program evaluation. Technical report submitted to the California Department of Social Services Office of Child Abuse Prevention.
7. Carrilio, T., Packard, T., & Clapp, J. (2003). Nothing in-nothing out: Barriers to the use of performance data in social service programs. Administration in Social Work, 27(4), 61-75.
8. Carrilio, T. (2003). Learning from experience? A review of three California initiatives addressing the needs of vulnerable families. Social Policy Journal, 2(2/3), 5-25.
9. Carrilio, T. (in press). The elusive search for the silver bullet in prevention and family support programs for vulnerable families. Social Work and Social Sciences Review.
10. Chafin, M. (2004). Is it time to re-think Healthy Start/Healthy Families? Child Abuse & Neglect, 28(6), 589-595.
11. Clapp, J. D., & Early, T. J. (1999). A qualitative exploratory study of substance abuse prevention outcomes in a heterogeneous prevention system. Journal of Drug Education, 29(3), 217-233.
12. Cournoyer, B. (2004). The evidence-based social work skills book. New York: Allyn & Bacon.
13. Daro. D., & Harding, K. (1999). Healthy Families America: Using research to enhance practice. In Behrman, R. (Ed.), The future of children: Recent program evaluations, 9(1). David and Lucille Packard Foundation, 152-176.
14. Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational & Psychological Consultation, 11(2), 93-221.
15. Donaldson, S. I, & Scriven, M. (2003). Evaluating social programs and problems. Mahwah, NJ: Lawrence Erlbaum Associates.
16. Duggan, A., McFarlane, E., Fuddy, L., Burrell, L., Higman, S. M., Winham, A., & Sia, C. (2004). Randomized trial of a statewide home visiting program: Impact in preventing child abuse and neglect. Child Abuse & Neglect, 28(6), 597-622.
17. Duggan, A., McFarlane, E., Windham, A., Rohde, C., Salkever, D., Fuddy, L., Rosenberg, L., Buchbinder, S., & Sia, C. (1999). Evaluation of Hawaii's Healthy Start Program in Behrman, R. (Ed.), The future of children: Recent program evaluations, 9(1). David and Lucille Packard Foundation, 66-90.
18. Eisenberg, D., & Carrilio, T. (1983). Informal resources for the elderly: Panacea or empty promise? Journal of Gerontological Social Work, 6(1), 39-47.
19. Erikson, M., & Kurz-Reimer, K. (1999). Infants, toddlers, and families: A framework for support and intervention. New York: The Guilford Press.
20. Everhart, K., & Wandersman, A. (2000). Applying comprehensive quality programming and empowerment evaluation to reduce implementation barriers. Journal of Educational & Psychological Consultation, 11(2), 177-191.
21. Gambrill, E.D. (2003). Evidence-based practice: Sea change or the emperor's new clothes? Journal of Social Work Education, 39(1), 3-23.
22. Garbarino, J. (1995). Raising children in a socially toxic environment. San Francisco: Jossey-Bass.
23. Gomby, D., Culross, P., & Behrman, R. (1999). Home visiting: Recent program evaluations - Analysis and recommendations. In Behrman, R. (Ed.), The future of children: Recent program evaluations, 9(1). David and Lucille Packard Foundation, 4-26.
24. Gomby, D. (1999). Understanding evaluations of home visitation programs. In Behrman, R. (Ed.), The future of children: Recent program evaluations, 9(1). David and Lucille Packard Foundation, 27-43.
25. Guterman, N. B. (2001). Stopping child maltreatment before it starts. Thousand Oaks, CA: Sage Publications.
26. Harachi, T. W., Abbott, R. D., Catalano, R. F., Haggerty, K. P., & Fleming, C. B. (1999). Opening the black box: Using process evaluation measures to assess implementation and theory building. American Journal of Community Psychology, 27(5), 711-731.
27. Hernandez, M. (2000). Using logic models and program theory to build outcome accountability. Education and Treatment of Children, 23(1), 24-40.
28. Hodge, D. R. (2003). Differences in worldviews between social workers and people of faith. Families in Society, 84(2), 285-295.
29. Hodges, S., & Hernandez, M. (1999). How organizational culture influences outcome information utilization. Evaluation and Program Planning, 22(2), 183-197.
30. Hogue, A. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33(2), 332-345.
31. Horsch, K. (1996). Results-based accountability systems: Opportunities and challenges. The Evaluation Exchange, 2(1), 2-3. Electronic journal published by The Harvard Family Research Project, retrieved on April 30, 2005 from http://www.gse.harvard.edu/hfrp/eval.html.
32. Illig, D. (1998). Birth to kindergarten: The importance of the early years. California State Library Monograph No. CR-98-001. Sacramento: California Research Bureau.
33. Jerrell, J., & Ridgely, M. (1999). The relative impact of treatment program robustness and dosage. Evaluation and Program Planning, 22(3), 323-330.
34. Julian, D., & Clapp, J. D. (2000). Outcomes based funding and the logic model: Mechanisms for coordinating human services at the local level. Evaluation and Program Planning, 23(2), 231-240.
35. Kagen, S. L., & Neville, P. (1993). Integrating services for children and families. New Haven: Yale University Press.
36. Karoly, L. Greenwood, P., Evringham, S., Hoube, J., Kilburn, M. R., Rydell, P., Saunders, M., & Chiesa, J. (1998). Investing in our children: What we know and don't know about the costs and benefits of early childhood interventions. Rand MR 898-TCWF. Santa Monica: Rand Corporation.
37. Knitzer, J. (1999). Promoting resiliency among the most stressed young children and families: Toward more responsive welfare-linked policies and programs. The National Center for Children in Poverty. Joseph L. Mailman School of Public Health. Columbia University.
38. Kotulak, R. (1995). Inside the brain: Revolutionary discoveries of how the mind works. Kansas City: Andrews and McMeel.
39. Krugman, R. (1993). Universal home visiting: A recommendation from the U.S. Advisory Board on Child Abuse and Neglect. In Behrman, R. (Ed.), The future of children: Home visiting, 3(3). David and Lucille Packard Foundation, 184-191.
40. Landsverk, J., Carrilio, T. E., Connelly, C. D., Ganger, W. C., Slymen, D. J., Newton, R. R., & Jones, C. (2002). Healthy Families San Diego clinical trial: Technical report. San Diego Children's Hospital and Health Center, Child and Adolescent Services Research Center.
41. Leff, H. S., & Mulkern, V. (2002). Lessons learned about science and participation from multisite evaluations. In J.M. Herrell & R.B. Straw (Eds.), New Directions for Evaluation, 94. San Francisco: Jossey-Bass, 89-100.
42. Marwell, N. P. (2004). Privatizing the welfare state: Nonprofit community-based organizations as political actors. American Sociological Review, 69(2), 265-291.
43. O'Connor, B. (2001). The protagonists and ideas behind the Personal Responsibility and Work Opportunity Reconciliation Act of 1996: The enactment of a conservative welfare system. Social Justice, 28(4), 4-32.
44. Olds, D., Henderson, C., Kitzman, H., Eckenrode, J., Cole, R., & Tattlebaum, R. (1999). Prenatal and infancy home visitation by nurses: Recent findings. In Behrman, R. (Ed.), The future of children: Recent program evaluations, 9(1). David and Lucille Packard Foundation, 44-63.
45. Olds, D. L., Robinson, J., O'Brien, R., Luckey, D. W., Pettit, L., Henderson, L., Jr., Ng, R., Sheff, K., Korfmacher, J., Hiat, S., & Talmi, A. (2002). Home visiting by para-professionals and by nurses: A randomized, controlled trial. Pediatrics, 110(3): 486-497.
46. Orwin, R. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95(supplement 13), S309-S327.
47. Pandey, S., & Collier-Tenison, S. (2001). Welfare reform: An exploration of devolution. Social Justice, 28(l), 54-75.
48. Poertner, J. (2000). Managing for service outcomes. In R. Patti (Ed.), The handbook of social welfare management. Thousand Oaks, CA: Sage Publications, 267-281.
49. Proctor, E. K. (2002). Quality of care and social work research. Social Work Research, 26(4), 195-197.
50. Reynolds, A. J., Temple, J., Robertson, D., & Mann, E. (2001). Long term effects of an early childhood intervention on educational achievement and juvenile arrest: A 15-year follow-up of low-income children in public schools. Journal of the American Medical Association, 285(18), 2339-2346.
51. Rossi, P. H., Freman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage Publications.
52. Schorr, L., & Schorr, D. (1988). Within our reach: Breaking the cycle of disadvantage. New York: Anchor Books (Doubleday).
53. Schorr, L. (1997). Common purpose: Strengthening families and neighborhoods to rebuild America. New York: Doubleday.
54. Shonkoff, J. P., & Phillips, D. A. (2000). From neurons to neighborhoods: The science of early childhood development. National Research Council and Institute of Medicine Board on Children, Youth, and Families, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
55. Simmel, C. (2002). The Shared Family Care development project: Challenges of implementing and evaluating a community based project. Children and Youth Services Review, 24(6-7),455-470.
56. Stratham, J., & Holtermann, S. (2004). Families on the brink: The effectiveness of family support services. Child & Family Social Work, 9(2), 153-166.
57. Straw, R. B., & Herrell, J. M. (2002). A framework for understanding and improving multisite evaluations. In J.M. Herrell & R.B. Straw (Eds.), New Directions for Evaluation, 94. San Francisco: Jossey-Bass: 5-15.
58. Teague, G. (1998). Program fidelity in assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68(2), 216-232.
59. Tidmarsh, J., Carpenter, J., & Slade, J. (2003). Practitioners as gatekeepers and researchers: Family support outcomes. The International Journal of Sociology and Social Policy, 13(1/2), 59-80.
60. Twombly, E. C. (2002). Religious versus secular human services organizations: Implications for public policy. Social Science Quarterly, 83(4), 947-961.
61. Vinson, N. (2001). The system of care model: Implementation in 27 communities. Journal of Emotional & Behavioral Disorders, 9(1), 30-42.
62. Webb, S. (2002). Evidence-based practice and decision analysis in social work: An implementation model. Journal of Social Work, 2(1),45-63.
63. Weiss, R. P. (2001). Charitable choice as neoliberal social welfare strategy. Social Justice, 28(1), 35-54.
64. Whitaker, I. P., & Time, V. (2001). Devolution and welfare: The social and legal implications of state inequalities for welfare reform in the United States. Social Justice, 28(1), 76-90.
65. Witkin, S. L., & Harrison, W. D. (2001) Whose evidence and for what purpose? Social Work, 46(4), 293-296.
66. Yoshikawa, H. (1995). Long-term effects of early childhood programs on social outcomes and delinquency. In Behrman, R. (Ed.), The future of children: Long term out-comes of early childhood programs. 5(3). David and Lucille Packard Foundation, 51-70.