Revisiting field experimentation: Field notes for the future

Psychological Methods

Volumn 7, Issue 1, 2002, Pages 3-18

  Shadish, William R ^a  

^a University of Memphis   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; FORECASTING; HUMAN; METHODOLOGY; PROGRAM DEVELOPMENT;

FORECASTING; HUMANS; PROGRAM DEVELOPMENT; RESEARCH DESIGN;

EID: 85047671883     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/1082-989X.7.1.3     Document Type: Review

References (128)

1. Abelson, R. P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science, 8, 12-15.
2. Allison, P. D. (2000). Multiple imputation for missing data: A cautionary tale. Sociological Methods & Research, 28, 301-309.
3. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
4. Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-455.
5. Aptech Systems. (2001). GAUSS: Mathematical and statistical system. Maple Valley, WA: Author.
6. Barcikowski, R. S. (1981). Statistical power with groups as the unit of analysis. Journal of Educational Statistics, 6, 267-285.
7. Beauchamp, T. L. (Ed.). (1974). Philosophical problems of causation. Encino, CA: Dickenson.
8. Bell, S. H., Orr, L. L., Blomquist, J. D., & Cain, G. G. (1995). Program applicants as a comparison group in evaluating training programs. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.
9. Berk, R. A., Smyth, G. K., & Sherman, L. W. (1988). When random assignment fails: Some lessons from the Minneapolis Spouse Abuse Experiment. Journal of Quantitative Criminology, 4, 209-223.
10. Bloom, H. S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8, 225-246.
11. Boruch, R. F. (1997). Randomized field experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.
12. Boruch, R. F., Dennis, M., & Carter-Greer, K. (1988). Lessons from the Rockefeller Foundation's experiments on the Minority Female Single Parent Program. Evaluation Review, 12, 396-426.
13. Boruch, R. F., & Wothke, W. (1985). Seven kinds of randomization plans for designing field experiments. In R. F. Boruch & W. Wothke (Eds.), Randomization and field experimentation (pp. 95-118). San Francisco: Jossey-Bass.
14. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
15. Bryk, A. S., Raudenbush, S. W., & Congdon, R. T. (1996). HLM: Hierarchical linear modeling with the HLM/2L and HLM/3L programs. (Available from Scientific Software International, 1525 E. 53rd Street. Suite 530, Chicago, IL 60615).
16. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297-312.
17. Campbell, D. T. (1969). Reforms as experiments. American Psychologist, 24, 409-429.
18. Campbell, D. T. (1971, September). Methods for the experimenting society. Paper presented at the annual meeting of the Eastern Psychological Association, New York, NY, and at the 79th Annual Convention of the American Psychological Association, Washington, DC.
19. Campbell, D. T. (1986). Relabeling internal and external validity for applied social scientists. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 67-77). San Francisco: Jossey-Bass.
20. Campbell, D. T. (1988). Methodology and epistemology for social science: Selected papers (E. S. Overman, Ed.). Chicago: University of Chicago Press.
21. Campbell, D. T. (1996). Regression artifacts in time-series and longitudinal data. Evaluation and Program Planning, 19, 377-389.
22. Campbell, D. T., & Boruch, R. F. (1975). Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In C. A. Bennett & A. A. Lumsdaine (Eds.), Evaluation and experiments: Some critical issues in assessing social programs (pp. 195-296). New York: Academic Press.
23. Campbell, D. T., & Kenny, D. A. (1999). A primer on regression artifacts. New York: Guilford Press.
24. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally.
25. Cochran, W. G. (1965). The planning of observational studies in human populations. Journal of the Royal Statistical Society, Series A, 128, 134-155.
26. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
27. Collins, L. M., & Sayer, A. G. (Eds.). (2001). New methods for the analysis of change. Washington, DC: American Psychological Association.
28. Cook, T. D. (1990). The generalization of causal connections Multiple theories in search of clear practice. In L. Sechrest, E. Perrin, & J. Bunker (Eds.), Research methodology: Strengthening causal interpretations of nonexperimental data (pp. 9-31, DHHS Publication No. PHS 90-3454). Rockville, MD: Department of Health and Human Services.
29. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand-McNally.
30. Cook, T. D., & Shadish, W. R. (1994). Social experiments: Some developments over the past 15 years. Annual Review of Psychology, 45, 545-580.
31. Cowles, M. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Erlbaum.
32. Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.
33. D'Agostino, R. B., & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95, 749-759.
34. Dawid, A. P. (2000). Causal inference without counterfactuals. Journal of the American Statistical Association, 95, 407-448.
35. Delucchi, K. L. (1994). Methods for the analysis of binary outcome results in the presence of missing data. Journal of Consulting and Clinical Psychology, 62, 569-575.
36. Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. London: Arnold.
37. Elkin, I., Shea, T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins, J. F., Glass, D. R., Pilkonis, P. A., Leber, W. R., Docherty, J. P., Fiester, S. J., & Parloff, M. B. (1989). National Institute of Mental Health Treatment of Depression Collaborative Research Program: General effectiveness of treatments. Archives of General Psychiatry, 46, 971-982.
38. Festinger, L. (1953). Laboratory experiments. In L. Festinger & D. Katz (Eds.), Research methods in the behavioral sciences (pp. 136-172). New York: Holt, Rinehart & Winston.
39. Festinger, L., Riecken, H. W. & Schachter, S. (1956). When prophecy fails. Minneapolis: University of Minnesota Press.
40. Finn, J. D., & Achilles, C. M. (1990). Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27, 557-577.
41. Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver and Boyd.
42. Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain, 33, 505-513.
43. Fisher, R. A. (1935). The design of experiments. Edinburgh, Scotland: Oliver and Boyd.
44. Greenberg, D., & Shroder, M. (1997). The digest of social experiments (2nd ed.). Washington, DC: The Urban Institute Press.
45. Greene, W. H. (1998). LIMDEP (Version 7. 0) [Computer software]. (Available from Econometric Software, 15 Gloria Place, Plainview, NY 11803)
46. Greene, W. H. (1999). Econometric analysis. Upper Saddle River, NJ: Prentice Hall.
47. Gueron, J. M. (1999, May). The politics of random assignment: Implementing studies and impacting policy. Paper presented at the Conference on Evaluation and Social Policy in Education, American Academy of Arts and Sciences, Harvard University, Cambridge, MA.
48. Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.
49. Hartmann, G. W. (1936). A field experiment on the comparative effectiveness of "emotional" and "rational" political leaflets in determining election results. Journal of Abnormal and Social Psychology, 31, 99-114.
50. Haveman, R. H. (1987). Poverty policy and poverty research: The Great Society and the social sciences. Madison: University of Wisconsin Press.
51. Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153-161.
52. Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64, 605-654.
53. Heckman, J. J., LaLonde, R. J., & Smith, J. A. (1999). The economics and econometrics of active labor market programs. In A. Ashenfelter & D. Card (Eds.), Handbook of labor economics (Vol. 3, pp. 1-160). Amsterdam: Elsevier Science.
54. Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-970.
55. Holland, P. W. (1989). Comment: It's very clear. Journal of the American Statistical Association, 84, 875-877.
56. Honaker, J., Joseph, A., King, G., Scheve, K., & Singh, N. (1999). Amelia: A program for missing data [Computer software]. Cambridge, MA: Harvard University. Retrieved from http://gking.harvard.edu/stats.shtml#amelia
57. Horton, N. J., & Lipsitz, S. R. (2001). Multiple imputation in practice: Comparison of software packages for regression models with missing variables. The American Statistician, 55, 244-254.
58. King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political science data: An alternative algorithm for multiple imputation. American Political Science Review, 95, 49-69.
59. Kraemer, H. C., & Thiemann, S. (1989). A strategy to use soft data effectively in randomized controlled clinical trials. Journal of Consulting and Clinical Psychology, 57, 148-154.
60. Kreft, I., & DeLeeuw, J. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
61. Kruskal, W., & Mosteller, F. (1979a). Representative sampling, I: Non-scientific literature. International Statistical Review, 47, 13-24.
62. Kruskal, W., & Mosteller, F. (1979b). Representative sampling, II: Scientific literature, excluding statistics. International Statistical Review, 47, 111-127.
63. Kruskal, W., & Mosteller, F. (1979c). Representative sampling, III: The current statistical literature. International Statistical Review, 47, 245-265.
64. Leaf, R. C., DiGiuseppe, R., Mass, R., & Alington, D. E. (1993). Statistical methods for analyses of incomplete service records: Concurrent use of longitudinal and cross-sectional data. Journal of Consulting and Clinical Psychology, 61, 495-505.
65. Lewin, K. (1948). Resolving social conflicts: Selected papers on group dynamics. New York: Harper & Row.
66. Lewis, D. (1973). Causation. Journal of Philosophy, 70, 556-567.
67. Littell, R. C., Milliken, G. A., Stroup, W. W., & Wolfinger, R. D. (1996). SAS system for mixed models. Cary, NC: SAS Institute.
68. Little, R. J., & Rubin, D. B. (1987). Statistical analysis with missing data. New York: Wiley.
69. Little, R. J., & Schenker, N. (1995). Missing data. In G. Arminger, C. C. Clogg, & M. E. Sobel (Eds.), Handbook of statistical modeling for the social and behavioral sciences (pp. 39-75). New York: Plenum Press.
70. Little, R. J., & Yau, L. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of noshows using Rubin's causal model. Psychological Methods, 3, 147-159.
71. Mackie, J. L. (1974). The cement of the universe: A study of causation. Oxford, England: Oxford University Press.
72. Maddock, J. E. & Rossi, J. S. (2001). Statistical power of articles published in three health-psychology related journals. Health Psychology, 20, 76-78.
73. Mark, M. M. (1986). Validity typologies and the logic and practice of quasi-experimentation. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 47-66). San Francisco: Jossey-Bass.
74. Maxwell, S. E. (1998). Longitudinal designs in randomized group comparisons: When will intermediate observations increase statistical power? Psychological Methods, 3, 275-290.
75. McCall, W. A. (1923). How to experiment in education. New York: MacMillan.
76. Mehta, P. D., & West, S. G. (2000). Putting the individual back into individual growth curves. Psychological Methods, 5, 23-43.
77. Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 4, 283-298.
78. Murray, D. M. (1998). Design and analysis of group-randomized trials. New York: Oxford University.
79. Muthen, B. O., Kaplan, D., & Hollis, M. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52, 431-462.
80. Neyman, J. (1935). Statistical problems in agricultural experimentation. Supplement of the Journal of the Royal Statistical Society, 2, 107-180.
81. Neyman, J. (1991). Applications of the theory of probability to agricultural experiments: Test of the principles (D. Dabrowska & T. Speed, Trans.). Statistical Science, 5, 463-480. (Reprinted from Roczniki Nauk Rolniczki, 10, 1-51, 1923).
82. Neyman, J. (1991). Applications of the theory of probability to agricultural experiments: Test of the principles (D. Dabrowska & T. Speed, Trans.). Statistical Science, 5, 463-480. (Reprinted from Roczniki Nauk Rolniczki, 10, 1-51, 1923).
83. Orr, L. L. (1999). Social experiments: Evaluating public programs with experimental methods. Thousand Oaks, CA: Sage.
84. Raudenbush, S. W. (2001). Comparing personal trajectories and drawing causal inferences from longitudinal data. Annual Review of Psychology, 52, 501-525.
85. Reichardt, C. S. (2000). A typology of strategies for ruling out threats to validity. In L. Bickman (Ed.), Research design: Donald Campbell's legacy (Vol. 2, pp. 89-115). Thousand Oaks, CA: Sage.
86. Reichardt, C. S., & Gollob, H. F. (1986). Satisfying the constraints of causal modeling. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 91-107). San Francisco: Jossey-Bass.
87. Reichardt, C. S., & Gollob, H. F. (1987). Taking uncertainty into account when estimating effects. In M. M. Mark & R. L. Shetland (Eds.), Multiple methods for program evaluation (pp. 7-22). San Francisco: Jossey-Bass.
88. Ribisl, K. M., Walton, M. A., Mowbray, C. T., Luke, D. A., Davidson, W. S., & Bootsmiller, B. J. (1996). Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations. Evaluation and Program Planning, 19, 1-25.
89. Rogosa, D. (1988). Myths about longitudinal research. In K. W. Schaie, R. T. Campbell, W. Meredith, & S. C. Rawlings (Eds.), Methodological issues in aging research (pp. 171-210). New York: Springer.
90. Rosenbaum, P. R. (1991). Discussing hidden bias in observational studies. Annals of Internal Medicine, 115, 901-905.
91. Rosenbaum, P. R. (1995). Observational studies. New York: Springer-Verlag.
92. Rosenbaum, P. R., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika. 70, 41-55.
93. Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. The Behavioral and Brain Sciences, 3, 377-386.
94. Rossi, P. H., & Lyall, K. C. (1978). An overview evaluation of the NIT experiment. In T. D. Cook, M. L. DelRosario, K. M. Hennigan, M. M. Mark, & W. M. K. Trochim (Eds.), Evaluation studies review annual (Vol. 3, pp. 412-428). Newbury Park, CA: Sage.
95. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688-701.
96. Rubin, D. B. (1986). Comment: Which it's have causal answers. Journal of the American Statistical Association, 81, 961-962.
97. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: Wiley.
98. Saunders, L. D., Irwig, L. M., Gear, J. S., & Ramushu, D. L. (1991). A randomized controlled trial of compliance improving strategies in Soweto hypertensives. Medical Care, 29, 669-678.
99. Schafer, J. L. (2000). NORM for Windows 95/98/NT [Computer software]. University Park, PA: Author. Retrieved from http://www.stat.psu.edu/~jls/misoftwa.html#top
100. Schafer, J. L., & Graham, J. W. (in press). Missing data: Our view of the state of the art. Psychological Methods.
101. Scharfstein, D. O., Rotnitzky, A., & Robins, J. M. (1999). Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94, 1096-1120.
102. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
103. Shadish, W. R. (2000). The empirical program of quasi-experimentation. In L. Bickman (Ed.), Validity and social experimentation: Donald Campbell's legacy (pp. 13-35). Thousand Oaks, CA: Sage.
104. Shadish, W. R., & Cook, T. D. (1999). Design rules: More steps towards a complete theory of quasi-experimentation. Statistical Science, 14, 294-300.
105. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental design for generalized causal inference. Boston: Houghton-Mifflin.
106. Shadish, W. R., Cook, T. D., & Houts, A. C. (1986). Quasi-experimentation in a critical multiplist mode. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 29-46). San Francisco: Jossey-Bass.
107. Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.
108. Shadish, W. R., Hu, X., Glaser, R. R., Kownacki, R. J., & Wong, T. (1998). A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes. Psychological Methods, 3, 3-22.
109. Shadish, W. R., Phillips, G., & Clark, M. H. (in press). Content and context: The impact of Campbell and Stanley. In R. J. Sternberg (Ed.), The anatomy of impact: What has made the great works of psychology great? Washington, DC: American Psychological Association.
110. Shih, W. J., & Quan, H. (1997). Testing for treatment differences with dropouts present in clinical trials - A composite approach. Statistics in Medicine, 16, 1225-1239.
111. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
112. Sommer, A., & Zeger, S. L. (1991). On estimating efficacy from clinical trials. Statistics in Medicine, 10, 45-52.
113. Stanton, M. D., & Shadish, W. R. (1997). Outcome, attrition and family-couples treatment for drug abuse: A meta-analysis and review of the controlled, comparative studies. Psychological Bulletin, 122, 170-191.
114. Statistical Solutions. (1998). SOLAS for Missing Data Analysis 1.0 [Computer software]. (Available from Statistical Solutions, 8 South Bank, Crosse's Green, Cork, Ireland)
115. Statistical Solutions. (1999). SOLAS for Missing Data Analysis 2.0 [Computer software]. (Available from Statistical Solutions, 8 South Bank, Crosse's Green, Cork, Ireland)
116. Statistical Solutions. (2001). SOLAS for Missing Data Analysis 3.0 [Computer software]. (Available from Statistical Solutions, 8 South Bank, Crosse's Green, Cork, Ireland)
117. Sullivan, C. M., Rumptz, M. H., Campbell, R., Eby, K. K., & Davidson, W. S. (1996). Retaining participants in longitudinal community research: A comprehensive protocol. Journal of Applied Behavioral Science, 32, 262-276.
118. Underwood, B. J. (1957). Psychological research. New York: Appleton-Century-Crofts.
119. Varnell, S., Murray, D. M., & Baker, W. L. (2001). An evaluation of analysts options for the one-group-per-condition design: Can any of the alternatives overcome the problems inherent in this design? Evaluation Review, 25, 440-453.
120. Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
121. Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. American Psychologist, 47, 1578-1585.
122. Welch, W. P., Frank, R. G., & Costello, A. J. (1983). Missing data in psychiatric research: A solution. Psychological Bulletin, 94, 177-180.
123. West, S. G., Biesanz, J., & Pitts, S. C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social psychology (pp. 40-84). New York: Cambridge University Press.
124. West, S. G., & Sagarin, B. (2000). Subject selection and loss in randomized experiments. In L. Bickman (Ed.), Research design: Donald Campbell's legacy (Vol. 2, pp. 117-154). Thousand Oaks, CA: Sage.
125. Wilkinson, L., & The Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
126. Willson, V. L., & Putnam, R. R. (1982). A meta-analysis of pretest sensitization effects in experimental design. American Educational Research Journal, 19, 249-258.
127. Winship, C., & Mare, R. D. (1992). Models for sample selection bias. Annual Review of Sociology, 18, 327-350.
128. Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659-707.