What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’

International Journal of Social Research Methodology

Volumn 19, Issue 4, 2016, Pages 481-490

  Gorard, Stephen ^a   Gorard, Jonathan ^b  

^a DURHAM UNIVERSITY   (United Kingdom)

^b KING'S COLLEGE LONDON   (United Kingdom)

Author keywords

attrition; evaluation; reliability; sampling; Significance testing

Indexed keywords

EID: 84945218747     PISSN: 13645579     EISSN: 14645300     Source Type: Journal    
DOI: 10.1080/13645579.2015.1091235     Document Type: Article

References (26)

1. L.Behaghel,, B.Crepon,, M.Gurgand,, & T.Le Barbanchon, (2009). Sample attrition bias in randomized surveys: A tale of two surveys (IZA Discussion Paper 4162). Retrieved July 6, 2014, from http://ftp.iza.org/dp4162.pdf
2. R.Berk,, & D.Freedman, (2001). Statistical assumptions as empirical commitments. Retrieved July 3, 2014, from http://www.stat.berkeley.edu/~census/berk2.pdf
3. I.Brunton-Smith,, J.Carpenter,, M.Kenward,, & R.Tarling, (2014). Multiple imputation for handling missing data in social research. Social Research Update, 65, 1–4.
4. R.Carver, (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378–399.10.17763/haer.48.3.t490261645281841
5. G.Cuddeback,, E.Wilson,, J.Orme,, & T.Combs-Orme, (2004). Detecting and statistically correcting sample selection bias. Journal of Social Service Research, 30, 19–33.10.1300/J079v30n03_02
6. G.Cumming, (2014). The new statistics: why and how. Psychological Science, 25, 1, 7–29.
7. P.Dolton,,, M.Lindeboom,, & G.Van den Berg, (2000). Survey attrition: A taxonomy and the search for valid instruments to correct for biases. Retrieved from http://www.fcsm.gov/99papers/berlin.html
8. R.Falk,, & C.Greenbaum, (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory and Psychology, 5, 75–98.10.1177/0959354395051004
9. S.Gorard, (2003). Quantitative methods in social science: The role of numbers made easy. London: Continuum.
10. S.Gorard, (2010a). All evidence is equal: The flaw in statistical reasoning. Oxford Review of Education, 36, 63–77.10.1080/03054980903518928
11. S.Gorard, (2010b). Measuring is more than assigning numbers. In G.Walford, E.Tucker, & M.Viswanathan (Eds.), The Sage handbook of measurement (pp. 389–408). Los Angeles, CA: Sage.10.4135/9781446268230
12. S.Gorard, (2014a). Confidence intervals, missing data and imputation. International Journal of Research in Educational Methodology, 5, 693–698.
13. S.Gorard, (2014b). A proposal for judging the trustworthiness of research findings. Radical Statistics, 110, 47–60.
14. S.Gorard, (2015). Rethinking “quantitative” methods and the development of new researchers. Review of Education, 3(1). 72–96
15. S.Gorard,, B.H.See,, & N.Siddiqui, (2014). Switch-on reading evaluation report. Retrieved from http://educationendowmentfoundation.org.uk/uploads/pdf/FINAL_EEF_Evaluation_Report_-_Switch-on_-_February_2014.pdf
16. S.Gorard,, N.Siddiqui,, & B.H.See, (2014). Response to intervention evaluation report. Retrieved from http://educationendowmentfoundation.org.uk/uploads/pdf/FINAL_EEF_Evaluation_Report_-_Response_to_Intervention_-_February_2014.pdf
17. M.Hansen,, & W.Hurwitz, (1946). The problem of non-response in sample surveys. Journal of the American Statistical Association, 41, 517–529.10.1080/01621459.1946.10501894
18. A.Kuhberger,, A.Fritz,, & T.Schemdl, (2014). Publication bias in psychology. PLoS one. doi:10.137/journal.pome.0105825
19. M.Lipsey,, K.Puzio,, C.Yun,, M.Hebert,, K.Steinka-Fry,, M.Cole,, … M.Busick, (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (pp. 1–46). Washington, DC: Institute of Education Sciences.
20. P.Meehl, (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.10.1086/phos.1967.34.issue-2
21. M.Pampaka,, G.Hutcheson,, & J.Williams, (2014). Handling missing data. International Journal of Research and Method in Education. doi:10.1080/1743727X.2014.979146
22. M.Peress, (2010). Correcting for survey nonresponse using variable response propensity. Journal of the American Statistical Association. Retrieved from http://www.rochester.edu/College/faculty/mperess/Nonresponse.pdf
23. K.Sheikh,, & S.Mattingly, (1981). Investigating non-response bias in mail surveys. Journal of Epidemiology and Community Health, 35, 293–296.10.1136/jech.35.4.293
24. S.Siegel, (1956). Nonparametric statistics for the behavioural sciences. Tokyo: McGraw Hill.
25. R.Slavin,, & D.Smith, (2009). The relationship between sample sizes and effect sizes in systematic reviews in education. Educational Evaluation and Policy Analysis, 31, 500–506.10.3102/0162373709352369
26. D.Watts, (1991). Why is introductory statistics difficult to learn? The American Statistician, 45, 290–291.