Understanding and misunderstanding randomized controlled trials

Social Science and Medicine

Volumn 210, Issue , 2018, Pages 2-21

  Deaton, Angus ^a,b,c   Cartwright, Nancy ^d,e  

^a PRINCETON UNIVERSITY   (United States)

^b NATIONAL BUREAU OF ECONOMIC RESEARCH   (United States)

^c UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^d DURHAM UNIVERSITY   (United Kingdom)

^e UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Balance; Bias; Economic development; External validity; Health; Precision; RCTs; Transportation of results

Indexed keywords

CONCEPTUAL FRAMEWORK; ECONOMIC DEVELOPMENT; ESTIMATION METHOD; PRECISION; PREDICTION; PUBLIC HEALTH;

CONTROLLED STUDY; CONVENIENCE SAMPLE; ECONOMIC DEVELOPMENT; EXTERNAL VALIDITY; HUMAN; MEDICINE; PREDICTION; RANDOMIZATION; RANDOMIZED CONTROLLED TRIAL; SCIENTIST; SOCIOLOGY; THEORETICAL STUDY; TRUST; COMPREHENSION; RANDOMIZED CONTROLLED TRIAL (TOPIC); STATISTICAL BIAS;

BIAS; COMPREHENSION; HUMANS; RANDOMIZED CONTROLLED TRIALS AS TOPIC;

EID: 85040234199     PISSN: 02779536     EISSN: 18735347     Source Type: Journal    
DOI: 10.1016/j.socscimed.2017.12.005     Document Type: Article

References (158)

1. Abdul Latif Jameel Poverty Action Lab, MIT. (2017). Retrieved August 21, 2017 from https://www.povertyactionlab.org/about-j-pal.
2. Academy of Medical Sciences, Sources of Evidence for Assessing the Safety, Efficacy, and Effectiveness of Medicines. 2017 Retrieved from https://acmedsci.ac.uk/file-download/86466482.
3. Aigner, D.J., The residential electricity time-of-use pricing experiments. What have we learned?. Wise, D.A., Hausman, J.A., (eds.) Social Experimentation, 1985, Chicago University Press for National Bureau of Economic Research, Chicago, Il, 11–54.
4. Angrist, J.D., Treatment effect heterogeneity in theory and practice. Econ. J. 114 (2004), C52–C83.
5. Angrist, J.D., Bettinger, E., Bloom, E., King, E., Kremer, M., Vouchers for private schooling in Colombia: evidence from a randomized natural experiment. Am. Econ. Rev. 92:5 (2002), 1535–1558.
6. Aron-Dine, A., Einav, L., Finkelstein, A., The RAND health insurance experiment, three decades later. J. Econ. Perspect. 27:1 (2013), 197–222.
7. Arrow, K.J., Two Notes on Inferring Long Run Behavior from Social Experiments. Document No. P-5546, 1975, Rand Corporation, Santa Monica, CA.
8. Ashenfelter, O., The labor supply response of wage earners. Palmer, J.L., Pechman, J.A., (eds.) Welfare in Rural Areas: the North Carolina–Iowa Income Maintenance Experiment, 1978, The Brookings Institution, Washington, DC, 109–138.
9. Attanasio, O., Meghir, C., Santiago, A., Education choices in Mexico: using a structural model and a randomized experiment to evaluate PROGRESA. Rev. Econ. Stud. 79:1 (2012), 37–66.
10. Attanasio, O., Cattan, S., Fitzsimons, E., Meghir, C., Rubio-Codina, M., Estimating the Production Function for Human Capital: Results from a Randomized Controlled Trial in Colombia. (Working Paper W15/06), 2015, Institute for Fiscal Studies, London.
11. Bacon, F., Novum organum. Ellis, R.L., Spedding, J., (eds.) The Philosophical Works of Francis Bacon, 1859, Longmans, London, England.
12. Bahadur, R.R., Savage, L.J., The non-existence of certain statistical procedures in nonparametric problems. Ann. Math. Stat. 25 (1956), 1115–1122.
13. Banerjee, A., Chassang, S., Montero, S., Snowberg, E., A Theory of Experimenters. (Working Paper 23867), 2017, National Bureau of Economic Research, Cambridge, MA.
14. Banerjee, A., Chassang, S., Snowberg, E., Decision Theoretic Approaches to Experiment Design and External Validity. (Working Paper 22167), 2016, National Bureau of Economic Research, Cambridge, MA.
15. Banerjee, A., Duflo, E., The experimental approach to development economics. Annu. Rev. Econ. 1 (2009), 151–178.
16. Banerjee, A., Duflo, E., Poor Economics: a Radical Rethinking of the Way to Fight Global Poverty. 2012, Public Affairs, New York, NY.
17. Banerjee, A., Duflo, E., Goldberg, N., Karlan, D., Osei, R., Parienté W., Shapiro, J., Thuysbaert, B., Udry, C., A multifaceted program causes lasting progress for the very poor: evidence from six countries. Science, 348(6236), 2015, 1260799.
18. Banerjee, A., Karlan, D., Zinman, J., Six randomized evaluations of microcredit: introduction and further steps. Am. Econ. J. Appl. Econ. 7:1 (2015), 1–21.
19. Bareinboim, E., Pearl, J., A general algorithm for deciding transportability of experimental results. J. Causal Inference 1:1 (2013), 107–134.
20. Bareinboim, E., Pearl, J., Transportability from multiple environments with limited experiments: completeness results. Welling, M., Ghahramani, Z., Cortes, C., Lawrence, N., (eds.) Advances in Neural Information Processing Systems, vol. 27, 2014, 280–288.
21. Bauchet, J., Morduch, J., Ravi, S., Failure vs displacement: why an innovative anti-poverty program showed no net impact in South India. J. Dev. Econ. 116 (2015), 1–16.
22. Bechtel, W., Discovering Cell Mechanisms: the Creation of Modern Cell Biology. 2006, Cambridge University Press, Cambridge, England.
23. Begg, C.B., Significance tests of covariance imbalance in clinical trials. Contr. Clin. Trials 11:4 (1990), 223–225.
24. Bhattacharya, D., Dupas, P., Inferring welfare maximizing treatment assignment under budget constraints. J. Econom. 167:1 (2012), 168–196.
25. Bitler, M.P., Gelbach, J.B., Hoynes, H.W., What mean impacts miss: distributional effects of welfare reform experiments. Am. Econ. Rev. 96:4 (2006), 988–1012.
26. Bleyer, A., Welch, H.G., Effect of three decades of screening mammography on breast-cancer incidence. N. Engl. J. Med. 367 (2012), 1998–2005.
27. Bloom, H.S., Hill, C.J., Riccio, J.A., Modeling cross-site experimental differences to find out why program effectiveness varies. Bloom, H.S., (eds.) Learning More from Social Experiments: Evolving Analytical Approaches, 2005, Russell Sage, New York, NY.
28. Bold, T., Kimenyi, M., Mwabu, G., Ng'ang’a, A., Sandefur, J., Scaling up what Works: Experimental Evidence on External Validity in Kenyan Education. (Working Paper 321), 2013, Center for Global Development, Washington, DC.
29. Bothwell, L.E., Podolsky, S.H., The emergence of the randomized, controlled trial. N. Engl. J. Med. 375:6 (2016), 501–504.
30. Cartwright, N., Nature's Capacities and Their Measurement. 1994, Clarendon Press, Oxford, England.
31. Cartwright, N., Single Case Causes: What Is Evidence and Why. 2015 (CHESS Working paper 2015-02). Retrieved from https://www.dur.ac.uk/resources/chess/CHESSWP_2015_02.pdf.
32. Cartwright, N., Hardie, J., Evidence Based Policy: a Practical Guide to Doing it Better. 2012, Oxford University Press, Oxford, England.
33. Chalmers, I., Comparing like with like: some historical milestones in the evolution of methods to create unbiased comparison groups in therapeutic experiments. Int. J. Epidemiol. 30 (2001), 1156–1164.
34. Concato, J., Study design and ‘evidence’ in patient-oriented research. Am. J. Respir. Crit. Care Med. 187:11 (2013), 1167–1172.
35. Concato, J., Shah, N., Horwitz, R.I., Randomized, controlled, trials, observational studies, and the hierarchy of research designs. N. Engl. J. Med. 342:25 (2000), 1887–1892.
36. Conlisk, J., Choice of response functional form in designing subsidy experiments. Econometrica 41:4 (1973), 643–656.
37. CONSORT, Baseline Data. 2010, 15 Retrieved November 9, 2017 from http://www.consort-statement.org/checklists/view/32–consort-2010/510-baseline-data.
38. Cook, T.D., Generating causal knowledge in the policy sciences: external validity as a task of both multi-attribute representation and multi-attribute extrapolation. J. Pol. Anal. Manag. 33:2 (2014), 527–536.
39. Craver, C., Explaining the Brain: Mechanisms and the Mosaic Unity of Neuroscience. 2007, Clarendon Press, Oxford, England.
40. Crépon, B., Duflo, E., Gurgand, M., Rathelot, R., Zamora, P., Do labor market policies have displacement effects? Evidence from a clustered randomized experiment. Q. J. Econ. 128:2 (2014), 531–580.
41. Davey-Smith, G., Ibrahim, S., Data dredging, bias, or confounding. Br. Med. J. 325 (2002), 1437–1438.
42. Dawid, A.P., Causal inference without counterfactuals. J. Am. Stat. Assoc. 95:450 (2000), 407–424.
43. Deaton, A., Instruments, randomization, and learning about development. J. Econ. Lit. 48:2 (2010), 424–455.
44. Deaton, A., Muellbauer, J., Economics and Consumer Behavior. 1980, Cambridge University Press, New York. NY.
45. Dhaliwal, I., Duflo, E., Glennerster, R., Tulloch, C., Comparative Cost-effectiveness Analysis to Inform Policy in Developing Countries: a General Framework with Applications for Education. December 3, 2012, Abdul Latif Jameel Poverty Action Lab, MIT Retrieved from http://www.povertyactionlab.org/publication/cost-effectiveness.
46. Duflo, E., Hanna, R., Ryan, S.P., Incentives work: getting teachers to come to school. Am. Econ. Rev. 102:4 (2012), 1241–1278.
47. Duflo, E., Kremer, M., Use of randomization in the evaluation of development effectiveness. Easterly, W., (eds.) Reinventing Foreign Aid, 2008, Brookings, Washington, DC, 93–120.
48. Dynarski, S., Helping the poor in education: the power of a simple nudge. N. Y. Times, 2015, January 18, BU6 Retrieved from https://www.nytimes.com/2015/01/18/upshot/helping-the-poor-in-higher-education-the-power-of-a-simple-nudge.html.
49. Epstein, S., Inclusion: the Politics of Difference in Medical Research. 2007, Chicago University Press, Chicago, Il.
50. Feinstein, A.R., Horwitz, R.I., Problems in the ‘evidence’ of ‘evidence-based medicine’. Am. J. Med. 103 (1997), 529–535.
51. Fine, P.E.M., Clarkson, J.A., Individual versus public priorities in the determination of optimal vaccination policies. Am. J. Epidemiol. 124:6 (1986), 1012–1020.
52. Fisher, R.A., The arrangement of field experiments. J. Minist. Agric. G. B. 33 (1926), 503–513.
53. Freedman, D.A., Statistical models for causation: what inferential leverage do they provide?. Eval. Rev. 30:6 (2006), 691–713.
54. Freedman, D.A., On regression adjustments to experimental data. Adv. Appl. Math. 40 (2008), 180–193.
55. Frieden, T.R., Evidence for health decision making—beyond randomized, controlled trials. N. Engl. J. Med. 377 (2017), 465–475.
56. Garfinkel, I., Manski, C.F., Introduction. Garfinkel, I., Manski, C.F., (eds.) Evaluating Welfare and Training Programs, 1992, Harvard University Press, Cambridge, MA, 1–22.
57. Gerber, A.S., Green, D.P., Field Experiments. 2012, Norton, New York, NY.
58. Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B., Vermeersch, C.M.J., Impact Evaluation in Practice. second ed., 2016, Inter-American Development Bank and World Bank, Washington, DC.
59. Greenberg, D., Shroder, M., Onstott, M., The social experiment market. J. Econ. Perspect. 13:3 (1999), 157–172.
60. Greenland, S., Randomization, statistics, and causal inference. Epidemiology 1:6 (1990), 421–429.
61. Greenland, S., Mansournia, M.A., Limitations of individual causal models, causal graphs, and ignorability assumptions, as illustrated by random confounding and design unfaithfulness. Eur. J. Epidemiol. 30 (2015), 1101–1110.
62. Gueron, J.M., Rolston, H., Fighting for Reliable Evidence. 2013, Russell Sage, New York, NY.
63. Guyatt, G., Sackett, D.L., Cook, D.J., Users’ guides to the medical literature II: how to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? For the Evidence-Based Medicine Working Group, 1994. J. Am. Med. Assoc. 271:1 (1994), 59–63.
64. Harrison, G.W., Field experiments and methodological intolerance. J. Econ. Meth. 20:2 (2013), 103–117.
65. Harrison, G.W., Impact evaluation and welfare evaluation. Eur. J. Dev. Res. 26 (2014), 39–45.
66. Harrison, G.W., Cautionary notes on the use of field experiments to address policy issues. Oxf. Rev. Econ. Pol. 30:4 (2014), 753–763.
67. Heckman, J.J., Randomization and social policy evaluation. Manski, C.F., Garfinkel, I., (eds.) Evaluating Welfare and Training Programs, 1992, Harvard University Press, Cambridge, MA, 547–570.
68. Heckman, J.J., Hohman, N., Smith, J., with the assistance of Khoo, M, Substitution and drop out bias in social experiments: a study of an influential social experiment. Q. J. Econ. 115:2 (2000), 651–694.
69. Heckman, J.J., Ichimura, H., Todd, P.E., Matching as an econometric evaluation estimator: evidence from evaluating a job training program. Rev. Econ. Stat. 64:4 (1997), 605–654.
70. Heckman, J.J., Lalonde, R.J., Smith, J.A., The economics and econometrics of active labor markets. Ashenfelter, O., Card, D., (eds.) Handbook of Labor Economics, 3A, 1999, North-Holland, Amsterdam, Netherlands, 1866–2097.
71. Heckman, J.J., Pinto, R., Savelyev, P., Understanding the mechanisms through which an influential early childhood program boosted adult outcomes. Am. Econ. Rev. 103:6 (2013), 2052–2086.
72. Heckman, J.J., Vytlacil, E.J., Econometric evaluation of social programs, Part 1: causal models, structural models, and econometric policy evaluation. Heckman, J.J., Leamer, E.E., (eds.) Handbook of Econometrics, 6B, 2007, North-Holland, Amsterdam, Netherlands, 4779–4874.
73. Hernán, M.A., A definition of a casual effect for epidemiological research. J. Epidemiol. Community Health 58:4 (2004), 265–271.
74. Hernán, M.A., Hernández-Diaz, S., Robins, J.M., Randomized trials analyzed as observational studies. Ann. Intern. Med. 159:8 (2013), 560–562.
75. Hill, A.B., The environment and disease: Association or causation?. Proc. Roy. Soc. Med. 58:5 (1965), 295–300.
76. Horton, R., Common sense and figures: the rhetoric of validity in medicine. Bradford Hill memorial lecture 1999. Stat. Med. 19 (2000), 3149–3164.
77. Horwitz, R.I., The dark side of evidence based medicine. Cleve. Clin. J. Med. 63:6 (1996), 320–323.
78. Horwitz, R.I., Hayes-Conroy, A., Caricchio, R., Singer, B.H., From evidence-based medicine to medicine-based evidence. Am. J. Med. 130:11 (2017), 1246–1250.
79. Hotz, V.J., Imbens, G.W., Mortimer, J.H., Predicting the efficacy of future training programs using past experience at other locations. J. Econom. 125 (2005), 241–270.
80. Howick, J., The Philosophy of Evidence-based Medicine. 2011, Wiley-Blackwell, Chichester, England.
81. Howick, J., Glasziou, J.P., Aronson, J.K., Problems with using mechanisms to solve the problem of extrapolation. Theor. Med. Bioeth. 34 (2013), 275–291.
82. Hsieh, C., Urquiola, M., The effects of generalized school choice on achievement and stratification: evidence from Chile's voucher program. J. Publ. Econ. 90 (2006), 1477–1503.
83. Humphreys, M., Jacobs, A., Qualitative Inference from Causal Models. 2017 Draft manuscript (version 0.2). Retrieved November 27, 2017 from http://www.columbia.edu/∼mh2245/qualdag.pdf.
84. Hurwicz, L., On the structural form of interdependent systems. Stud. Logic Found. Math. 44 (1966), 232–239.
85. Ilardi, S.S., Craighead, W.E., Rapid early response, cognitive modification, and nonspecific factors in cognitive behavior therapy for depression: a reply to Tang and DeRubeis. Clin. Psychol. Sci. Pract. 6 (1999), 295–299.
86. Imbens, G.W., Nonparametric estimation of average treatment effects under exogeneity: a review. Rev. Econ. Stat. 86:1 (2004), 4–29.
87. Imbens, G.W., Kolesár, M., Robust standard errors in small samples: some practical advice. Rev. Econ. Stat. 98:4 (2016), 701–712.
88. Imbens, G.W., Wooldridge, J.M., Recent developments in the econometrics of program evaluation. J. Econ. Lit. 47:1 (2009), 5–86.
89. International Committee of Medical Journal Editors, Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. 2015 Retrieved August 20, 2016 from http://www.icmje.org/icmje-recommendations.pdf.
90. Kahneman, D., Klein, G., Conditions for intuitive expertise: a failure to disagree. Am. Psychol. 64:6 (2009), 515–526.
91. Karlan, D., Appel, J., More than Good Intentions: How a New Economics Is Helping to Solve Global Poverty. 2011, Dutton, New York, NY.
92. Kasy, M., Why experimenters might not want to randomize, and what they could do instead. Polit. Anal. 24:3 (2016), 324–338, 10.1093/pan/mpw012.
93. Kramer, P., Ordinarily Well: the Case for Antidepressants. 2016, Farrar, Straus, and Giroux, New York, NY.
94. Kramer, U., Stiles, W.B., The responsiveness problem in psychotherapy: a review of proposed solutions. Clin. Psychol. Sci. Pract. 22:3 (2015), 277–295.
95. Kremer, M., Holla, A., Improving education in the developing world: what have we learned from randomized evaluations?. Annu. Rev. Econ. 1 (2009), 513–542.
96. Lakatos, I., Falsification and the methodology of scientific research programmes. Lakatos, I., Musgrave, A., (eds.) Criticism and the Growth of Knowledge: Proceedings of the International Colloquium in the Philosophy of Science, London, 1970, Cambridge University Press, Cambridge, England, 91–106, 10.1017/CBO9781139171434.00.
97. Lalonde, R.J., Evaluating the econometric evaluations of training programs with experimental data. Am. Econ. Rev. 76:4 (1986), 604–620.
98. Lehman, E.L., Romano, J.P., Testing Statistical Hypotheses. third ed., 2005, Springer, New York, NY.
99. LeLorier, J., Grégoire, G., Benhaddad, A., Lapierre, J., Derderian, F., Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N. Engl. J. Med. 337 (1997), 536–542.
100. Levy, S., Progress against Poverty: Sustaining Mexico's Progresa-oportunidades Program. 2006, Brookings, Washington, DC.
101. Little, D., Across the Boundaries: Extrapolation in Biology and Social Science. 2007, Oxford University Press, Oxford, England.
102. Longford, N.T., Nelder, J.A., Statistics versus statistical science in the regulatory process. Stat. Med. 18 (1999), 2311–2320.
103. Machamer, P., Darden, L., Craver, C., Thinking about mechanisms. Philos. Sci. 67 (2000), 1–25.
104. Mackie, J.L., The Cement of the Universe: a Study of Causation. 1974, Oxford University Press, Oxford, England.
105. Manning, W.G., Newhouse, J.P., Duan, N., Keeler, E., Leibowitz, A., Health insurance and the demand for medical care: evidence from a randomized experiment. Am. Econ. Rev. 77:3 (1987), 251–277.
106. Manning, W.G., Newhouse, J.P., Duan, N., Keeler, E., Benjamin, B., Leibowitz, A., Marquis, M.A., Zwanziger, J., Health Insurance and the Demand for Medical Care: Evidence from a Randomized Experiment. 1988, RAND, Santa Monica, CA.
107. Manski, C.F., Treatment rules for heterogeneous populations. Econometrica 72:4 (2004), 1221–1246.
108. Manski, C.F., Public Policy in an Uncertain World: Analysis and Decisions. 2013, Harvard University Press, Cambridge, MA.
109. Manski, C.F., Tetenov, A., Sufficient trial size to inform clinical practice. PNAS 113:38 (2016), 10518–10523.
110. Marmot, M.G., Altman, D.G., Cameron, D.A., Dewar, J.A., Thomson, S.G., Wilcox, M., the Independent UK panel on breast cancer screening, The benefits and harms of breast cancer screening: an independent review. Br. J. Canc. 108:11 (2013), 2205–2240.
111. Metcalf, C.E., Making inferences from controlled income maintenance experiments. Am. Econ. Rev. 63:3 (1973), 478–483.
112. Moffitt, R., The labor supply response in the Gary experiment. J. Hum. Resour. 14:4 (1979), 477–487.
113. Moffitt, R., Evaluation methods for program entry effects. Manski, C., Garfinkel, I., (eds.) Evaluating Welfare and Training Programs, 1992, Harvard University Press, Cambridge, MA, 231–252.
114. Morgan, K.L., Rubin, D.B., Rerandomization to improve covariate balance in experiments. Ann. Stat. 40:2 (2012), 1263–1282.
115. Muller, S.M., Causal interaction and external validity: obstacles to the policy relevance of randomized evaluations. World Bank Econ. Rev. 29 (2015), S217–S225.
116. Orcutt, G.H., Orcutt, A.G., Incentive and disincentive experimentation for income maintenance policy purposes. Am. Econ. Rev. 58:4 (1968), 754–772.
117. Parkkinen, V.-P., Wallmann, C., Wilde, M., Clarke, B., Illari, P., Kelly, M.P., Norrell, C., Russo, F., Shaw, B., Williamson, J., Evaluating Evidence of Mechanisms in Medicine: Principles and Procedures. 2008, Springer, New York, NY.
118. Patsopoulos, N.A., A pragmatic view on pragmatic trials. Dialogues Clin. Neurosci. 13:2 (2011), 217–224.
119. Pearl, J., Bareinboim, E., Transportability of causal and statistical relations: a formal approach. Proceedings of the 25th AAAI Conference on Artificial Intelligence, 2011, AAAI Press, Menlo Park, CA, 247–254.
120. Pearl, J., Bareinboim, E., External validity: from do-calculus to transportability across populations. Stat. Sci. 29:4 (2014), 579–595.
121. Pitman, S.R., Hilsenroth, M.J., Goldman, R.E., Levy, S.R., Siegel, D.F., Miller, R., Therapeutic technique of APA master therapists: areas of difference and integration across theoretical orientations. Prof. Psychol. Res. Pract. 48:3 (2017), 156–166.
122. Rawlins, M., De testimonio: on the evidence for decisions about the use of therapeutic interventions. Lancet 372 (2008), 2152–2161.
123. Reichenbach, H., Nomological Statements and Admissible Operations. 1954, North-Holland, Amsterdam, Netherlands.
124. Reichenbach, H., Laws, Modalities and Counterfactuals, with a Foreword by W.C. Salmon. 1976, CA, University of California Press, Berkeley and Los Angeles.
125. Reiss, J., Against External Validity. 2017 (CHESS Working Paper 2017-03). Retrieved from https://www.dur.ac.uk/resources/chess/CHESSK4UWP_2017_03_Reiss.pdf.
126. Rothman, K.J., Causes. Am. J. Epidemiol. 104:6 (1976), 587–592.
127. Rothman, K.J., Epidemiology. An Introduction. second ed., 2012, Oxford University Press, New York, NY.
128. Rothman, K.J., Gallacher, J.E.J., Hatch, E.E., Why representativeness should be avoided. Int. J. Epidemiol. 42:4 (2013), 1012–1014.
129. Rothwell, P.M., External validity of randomized controlled trials: ‘To whom do the results of the trial apply’. Lancet 365 (2005), 82–93.
130. Rubin, D.B., Causal inference using potential outcomes: design, modeling, decisions. 2004 Fisher Lecture. J. Am. Stat. Assoc. 100:469 (2005), 322–331.
131. Russell, B., The Problems of Philosophy. 1912, Arc Manor, Rockville, MD (2008).
132. Sackett, D.L., Rosenberg, W.M.C., Gray, J.A.M., Haynes, R.B., Richardson, W.S., Evidence based medicine: what it is and what it isn't. Br. Med. J. 312 (1996), 71–72.
133. Savage, L.J., Subjective probability and statistical practice. Barnard, G.A., Cox, G.A., (eds.) The Foundations of Statistical Inference, 1962, Methuen, London, England, 9–35.
134. Scriven, M., Evaluation perspectives and procedures. Popham, W.J., (eds.) Evaluation in Education—current Applications, 1974, McCutchan Publishing Corporation, Berkeley, CA, 68–84.
135. Seckinelgin, H., The Politics of Global AIDS: Institutionalization of Solidarity, Exclusion of Context. Social Aspects of HIV, 3, 2017, Springer International Publishing, Switzerland.
136. Senn, S., Seven myths of randomization in clinical trials. Stat. Med. 32 (2013), 1439–1450.
137. Shadish, W.R., Cook, T.D., Campbell, D.T., Experimental and Quasi-experimental Designs for Generalized Causal Inference. 2002, Houghton Mifflin, Boston, MA.
138. Shepherd, G.M., Neurobiology. second ed., 1988, Oxford University Press, New York, NY.
139. Stuart, E.A., Cole, S.R., Bradshaw, C.P., Leaf, P.J., The use of propensity scores to assess the generalizability of results from randomized trials. J. Roy. Stat. Soc. A 174:2 (2011), 369–386.
140. Student (Gosset, W. S.), Comparison between balanced and random arrangements of field plots. Biometrika 29:3/4 (1938), 363–378.
141. Suzuki, E., Yamamoto, E., Tsuda, T., Identification of operating mediation and mechanism in the sufficient-component cause framework. Eur. J. Epidemiol. 26 (2011), 347–357.
142. Svorencik, A., The Experimental Turn in Economics: a History of Experimental Economics., 2015, Utrecht School of Economics Dissertation Series #29. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2560026.
143. Todd, P.E., Wolpin, K.J., Assessing the impact of a school subsidy program in Mexico: using a social experiment to validate a dynamic behavioral model of child schooling and fertility. Am. Econ. Rev. 96:5 (2006), 1384–1417.
144. Todd, P.E., Wolpin, K.J., Ex ante evaluation of social programs. Ann. d'Econ. Stat. 91/92 (2008), 263–291.
145. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Identifying and Implementing Educational Practices Supported by Rigorous Evidence: a User Friendly Guide. 2003, Institute of Education Sciences, Washington, DC.
146. Van der Weele, T.J., Confounding and effect modification: distribution and measure. Epidemiol. Meth. 1 (2012), 55–82.
147. Vandenbroucke, J.P., When are observational studies as credible as randomized controlled trials?. Lancet 363 (2004), 1728–1731.
148. Vandenbroucke, J.P., The HRT controversy: observational studies and RCTs fall in line. Lancet 373 (2009), 1233–1235.
149. Vittengl, J.R., Clark, L.A., Thase, M.E., Jarrett, R.B., Are improvements in cognitive content and depressive symptoms correlates or mediators during acute- phase cognitive therapy for recurrent major depressive disorder?. Int. J. Cognit. Ther. 7:3 (2014), 255–271, 10.1521/ijct.2014.7.3.251.
150. Vivalt, E., How Much Can We Generalize from Impact Evaluations?. 2016, NYU unpublished. Retrieved from http://evavivalt.com/wp-content/uploads/2014/12/Vivalt_JMP_latest.pdf (retrieved, Nov. 28, 2017).
151. Williams, H.C., Burden-Teh, E., Nunn, A.J., What is a pragmatic clinical trial?. J. Invest. Dermatol. 135:6 (2015), 1–3.
152. Wise, D.A., A behavioral model versus experimentation: the effects of housing subsidies on rent. Brucker, P., Pauly, R., (eds.) Methods of Operations Research, vol. 50, 1985, Anton Hain, Verlag, Germany, 441–489.
153. Wolpin, K.I., The Limits of Inference without Theory. 2013, MIT Press, Cambridge, MA.
154. Worrall, J., Evidence in medicine and evidence-based medicine. Philos. Compass 2/6 (2007), 981–1022.
155. Worrall, J., Evidence and ethics in medicine. Perspect. Biol. Med. 51:3 (2008), 418–431.
156. Yates, F., The comparative advantages of systematic and randomized arrangements in the design of agricultural and biological experiments. Biometrika 30:3/4 (1939), 440–466.
157. Young, A., Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results (Working Paper). 2017, London School of Economics Retrieved from http://personal.lse.ac.uk/YoungA/ChannellingFisher.pdf.
158. Ziliak, S.T., Balanced versus randomized field experiments in economics: why W. S. Gosset aka ‘Student’ matters. Rev. Behav. Econ. 1 (2014), 167–208.