Insight Into Evaluation Practice: A Content Analysis of Designs and Methods Used in Evaluation Studies Published in North American Evaluation-Focused Journals

American Journal of Evaluation

Volumn 31, Issue 3, 2010, Pages 326-346

  Christie, Christina A ^a   Fleischer, Dreolin Nesbitt ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b CLAREMONT GRADUATE UNIVERSITY   (United States)

Author keywords

content analysis; design; methods; scientifically based evaluation

Indexed keywords

EID: 77956631227     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214010369170     Document Type: Article

References (45)

1. Alkin, M. C., & Christie, C. A. (Eds.). (2005). Theorists' models in action [Special Issue]. New Directions for Evaluation, 106, 1-3.
2. American Evaluation Association. (2008, April). American Evaluation Association Internal Scan Report to the Membership, by Goodman Research Group. Retrieved September 15, 2008, from www.eval.org.
3. American Evaluation Association (2009, December). AEA outstanding evaluation award. Retrieved from http://www.eval.org/aboutus/awards/callforawards/aea.OutstandingEval.asp.
4. Azzam, T., & Christie, C.A. (2007). Using public databases to study relative program impact. Canadian Journal of Program Evaluation, 22, 57-68.
5. Buboltz, W.C., Jr., Miller, M., & Williams, D.J. (1999). Content analysis of research in the Journal of Counseling Psychology (1973-1998). Journal of Counseling Psychology, 46, 496-503.
6. Chatterji, M. (2007). Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. American Journal of Evaluation, 28, 239-255. doi: 10.1177/1098214007304884.
7. Christie, C.A. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. In C. A. Christie (Ed.), The practice-theory relationship in evaluation. New Directions for Evaluation, 97, doi: 10.1002/ev.72.
8. Christie, C.A. (2008). Analyzing the practice of evaluation: What do these cases tell us about theory? In J. Fitzpatrick, C. Christie, & M. Mark (Eds.), Evaluation in Action: Interviews with expert evaluators. Thousand Oaks, CA: SAGE.
9. Dillon, J.T. (1983). The use of questions in educational research. Educational Researcher, 12, 19-24.
10. Donaldson, S. I., Christie, C. A., & Mark, M. M. (Eds.). (2009). What counts as credible evidence in applied research and evaluation practice? Thousand Oaks, CA: SAGE.
11. Fetterman, D. M., & Wandersman, A. (Eds.). (2004). Empowerment evaluation principles in practice. New York, NY: Guilford Press.
12. Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2003). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Allyn & Bacon.
13. Fleischer, D., & Christie, C.A. (2009). Evaluation use: A survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30, 158-175. doi: 10.1177/1098214008331009.
14. Franco, S. (2007). Reauthorization of NCLB: Time to reconsider the scientifically based research requirement. Nonpartisan Education Review, Essays, 3, 1-14.
15. Greene, J.C. (2001). Mixing social inquiry methodologies. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 251-258). Washington, DC: American Educational Research Association.
16. Greene, J.C. (2002). With a splash of soda, please: Towards active engagement with difference. Evaluation, 8, 259-266. doi: 10.1177/1358902002008002522.
17. Greene, J.C. (2005a). Synthesis: A reprise on mixing methods. In T. S. Weisner (Ed.), Discovering successful pathways in children's development: Mixed methods in the study of childhood and family life (pp. 405-419). Chicago, IL: University of Chicago Press.
18. Greene, J.C. (2005b). Evaluators as stewards of the public good. In S. Hood, R. K. Hopson, & H. T. Frierson (Eds.), The role of culture and cultural context: A mandate for inclusion, truth, and understanding in evaluation theory and practice, Evaluation and Society Series (pp. 7-20). Greenwich, CT: Information Age Publishing.
19. Greene, J.C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13, 93-99.
20. Greene, J.C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.
21. Greene, J.C., & Caracelli, V.J. (1997a). Defining and describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 74. San Francisco, CA: Jossey-Bass. doi: 10.1002/ev.1068.
22. Greene, J. C., & Caracelli, V. J. (Eds.). (1997b). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 74. San Francisco, CA: Jossey-Bass.
23. Greene, J.C., & Caracelli, V.J. (2003). Making paradigmatic sense of mixed methods practice. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 91-110). Thousand Oaks, CA: SAGE.
24. Greene, J.C., Caracelli, V.J., & Graham, W.F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274. doi: 10.3102/ 01623737011003255.
25. Gribbons, B., & Herman, J. (1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5. Retrieved December 28, 2009, fromhttp://PAREonline.net/getvn.asp?v=5&n=14.
26. Jarvelin, K., & Vakkari, P. (1993). The evolution of library and information science 1965-1985: A content analysis of journal articles. Information Processing & Management, 29, 129-144.
27. Julnes, G., & Rog, D.J. (2007). Pragmatic support for policies on methodology. In G. Julnes & D. J. Rog (Eds.), Informing federal policies on evaluation methodology: Building the evidence base for method choice in government sponsored evaluation. New Directions for Evaluation, 113, 129-147. doi: 10.1002/ev.219.
28. King, J.A., & Stevahn, L. (in press). Interactive evaluation in practice: Managing the interpersonal dynamics of program evaluation. Thousand Oaks, CA: SAGE.
29. Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Beverly Hills, CA: SAGE.
30. Miller, R.L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296-319. doi: 10.1177/109821400602700303.
31. Neuendorf, K.A. (2001). The Content Analysis Guidebook. Thousand Oaks, CA: SAGE.
32. Patton, M.Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: SAGE.
33. Reichardt, C. S., & Cook, T. D. (Eds.). (1979). Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: SAGE.
34. Reysen, S. (2006). Publication of nonsignificant results: A survey of psychologists' opinions. Psychological Reports, 98, 169-175. doi: 10.2466/PR0.98.1.169-175.
35. Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W.H., & Shavelson, R.J. (2007). Estimating causal effects using experimental and observational designs (report from the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association.
36. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
37. Shadish, W.R., Cook, T.D., & Leviton, L.C.L. (1991). Foundations of program evaluation. Newbury Park, CA: SAGE.
38. Shavelson, R.J., & Towne, L. (2004). What drives scientific research in education: Questions, not methods, should drive the enterprise. APS Observer, 17, 27-30.
39. Smith, N.L., & Mukherjee, P. (1994). Classifying research questions addressed in published evaluation studies. Educational Evaluation and Policy Analysis, 16, 223-230.
40. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE.
41. Trenholm, C., Devaney, B., Fortson, K., Clark, M., Quay, L., & Wheeler, J. (2008). Impacts of abstinence education on teen sexual activity, risk of pregnancy, and risk of sexually transmitted diseases. Journal of Policy Analysis and Management, 27, 255-276. doi: 10.1002/pam.20324.
42. U. S. Department of Education. (2003). Scientifically based evaluation methods (RIN 1890-ZA00). Federal Register, 68, 62445-62447.
43. U. S. Department of Education. (2005). Scientifically based evaluation methods (RIN 1890-ZA00). Federal Register, 70, 3586-3589.
44. Webb, E., Campbell, D., Schwartz, R., & Sechrest, L. (1966). Unobtrusive measures: Nonreactive research in the social sciences. Chicago, IL: Rand McNally.
45. Weiss, C.H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.