Caveats for using statistical significance tests in research assessments

Journal of Informetrics

Volumn 7, Issue 1, 2013, Pages 50-62

  Schneider, Jesper W ^a  

^a AARHUS UNIVERSITY   (Denmark)

Author keywords

Problems and misuse; Research assessment; Statistical significance tests

Indexed keywords

RESEARCH;

CONFIDENCE INTERVAL; CRITICAL THINKING; NULL HYPOTHESIS; PROBLEMS AND MISUSE; SCIENTOMETRICS; STANDARD ERRORS; STATISTICAL SIGNIFICANCE TEST; WHETHER DIFFERENCES;

STATISTICS;

EID: 84867470426     PISSN: 17511577     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.joi.2012.08.005     Document Type: Review

References (102)

1. Abelson R.P. A retrospective on the significance test ban of 1999 (if there were no significance tests, they would be invented). What if there were no significance tests? 1997, 117-141. Erlbaum, Hillsdale, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
2. Abelson R.P. On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science 1997, 8(1):12-15.
3. Adler R., Ewing J., Taylor P. Citation statistics. Statistical Science 2009, 24(1):1-14.
4. American Psychological Association Publication manual of the APA 2010, APA, Washington, DC. 6th ed.
5. Anderson D.R. Model based inference in the life sciences. A primer on evidence 2008, Springer, New York, NY.
6. Anderson D.R., Burnham K.P., Thompson W.L. Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management 2000, 64:912-923.
7. Armstrong J.S. Significance tests harm progress in forecasting. International Journal of Forecasting 2007, 23(2):321-327.
8. Bakan D. The test of significance in psychological research. Psychological Bulletin 1966, 66(6):423-437.
9. Berk R.A., Freedman D.A. Statistical assumptions as empirical commitments. Law, punishment, and social control: Essays in honor of Sheldon Messinger 2003, 235-254. Aldine, New York. T.G. Blomberg, S. Cohen (Eds.).
10. Berk R.A., Western B., Weiss R.E. Statistical inference for apparent populations. Sociological Methodology 1995, 25:421-458.
11. Berk R.A., Western B., Weiss R.E. Reply to Bollen, Firebaugh, and Rubin. Sociological Methodology 1995, 25:481-485.
12. Berkson J. Some difficulties of interpretation encountered in the application of the chi-square test. Journal of the American Statistical Association 1938, 33(203):526-536.
13. Berkson J. Tests of significance considered as evidence. Journal of the American Statistical Association 1942, 37(219):325-335.
14. Bollen K.A. Apparent and nonapparent significance tests. Sociological Methodology 1995, 25:268-459.
15. Bornmann L. Towards an ideal method of measuring research performance: Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics 2010, 4(3):441-443.
16. Carver R.P. The case against statistical significance testing. Harvard Educational Review 1978, 48(3):378-399.
17. Chow S.L. Précis of statistical significance: Rationale, validity, and utility. Behavioral and Brain Sciences 1998, 2:169-239.
18. Cochran W.G. Sampling techniques 1953, Wiley, New York.
19. Cohen J. Statistical power analysis for the behavioral sciences 1988, Lawrence Erlbaum, Hillsdale, NJ. 2nd edition.
20. Cohen J. Things I have learned (so far). American Psychologist 1990, 45(12):1304-1312.
21. Cohen The earth is round (p<.05). American Psychologist 1994, 49(12):1003-1007.
22. Colliander C., Ahlgren P. The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments. Journal of Informetrics 2011, 5(1):101-113.
23. Copas J.B., Li H.G. Inference for non-random samples (with discussion). Journal Royal Statistical Society, B 1997, 59(1):55-95.
24. Cortina J.M., Dunlap W.P. On the logic and purpose of significance testing. Psychological Methods 1997, 2(2):161-172.
25. Coulson M., Healey M., Fidler F., Cumming G. Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Frontiers in Psychology 2010, 1:1-9.
26. Cox D.R. Principles of statistical inference 2006, Cambridge University Press, Cambridge, UK.
27. Cox D.R., Spjøtvoll E., Johansen S., van Zwet W.R., Bithell J.F., Barndorff-Nielsen O., et al. The role of significance tests [with discussion and reply]. Scandinavian Journal of Statistics 1977, 4(2):49-70.
28. Cumming G. Understanding the new statistics. Effect sizes, confidence intervals, and meta-analysis 2012, Routledge, New York.
29. Cumming G., Fidler F., Vaux D.L. Error bars in experimental biology. Journal of Cell Biology 2007, 177(1):7-11.
30. Diaconis P., Efron B. Computer-intensive methods in statistics. Scientific American 1983, 248(5):116-130.
31. Dines Z. Understanding psychology as a science: An introduction to scientific and statistical inference 2008, Palgrave Macmillan, Houndsmills, Basigstoke, UK.
32. Ellis P.D. The essential guide to effect sizes: Statistical power, meta-analysis and the interpretation of research results 2010, Cambridge University Press, Cambridge, UK.
33. Fidler F., Cumming G. Lessons learned from statistical reform efforts in other disciplines. Psychology in the Schools 2007, 44(5):441-449.
34. Fisher R.A. Statistical methods for research workers 1925, Oliver & Boyd, Edinburgh.
35. Freedman D.A. Sampling. Encyclopedia of social science research methods 2003, Retrieved December 01, 2011, from http://sage-ereference.com/view/socialscience/n879.xml.
36. Gelman A., Carlin J.B., Stern H.S., Rubin D.B. Bayesian data analysis 2004, Chapman & Hall/CRC, Boca Raton.
37. Gigerenzer G. The superego, the ego and the id in statistical reasoning. A handbook for data analysis in the behavioral sciences: Methodological issues 1993, 311-339. Erlbaum, Hillsdale, NJ. G. Keren, C. Lewis (Eds.).
38. Gigerenzer G. Mindless statistics. The Journal of Socio-Economics 2004, 33(5):587-606.
39. Gigerenzer G., Swijtink Z., Porter T., Daston L., Beatty J., Kruger L. The empire of chance: How probability changed science and everyday life 1989, Cambridge University Press, New York.
40. Gingras Y., Larivière V. There are neither king nor crown in scientometrics: Comments on a supposed alternative method of normalization. Journal of Informetrics 2011, 5(1):226-227.
41. Goodman S.N. P values, hypothesis tests, and likelihood: Implications for epidemiology of a neglected historical debate. American Journal of Epidemiology 1993, 137(5):485-496.
42. Goodman S.N. A dirty dozen: Twelve p-value misconceptions. Seminars in Hematology 2008, 45(3):135-140.
43. Greenland S. Randomization, statistics, and causal inference. Epidemiology 1990, 1(6):421-429.
44. Grissom R.J., Kim J.J. Effect sizes for research: A broad practical approach 2005, Lawrence Erlbaum, Mahwah, NJ.
45. Guttman L. The illogic of statistical inference for cumulative science. Applied Stochastic Models and Data Analysis 1985, 1(1):3-10.
46. Haslam N., Ban L., Kaufmann L., Loughnan S., Peters K., Whelan J., et al. What makes an article influential? Predicting impact in social and personality psychology. Scientometrics 2008, 76(1):169-185.
47. Huberty C.J., Pike C.J. On some history regarding statistical testing. Advances in social science methodology 1999, Vol. 5:1-22.
48. Hunter J.E. Needed: A ban on the significance test. Psychological Science 1997, 8:3-7.
49. Jacob J., Lehrl S., Henkel A. Early recognition of high quality researchers of the German psychiatry by worldwide accessible bibliometric indicators. Scientometrics 2007, 73(2):117-130.
50. Keselman H.J., Huberty C.J., Lix L.M., Olejnik S., Cribbie R.A., Donahue B., et al. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research 1998, 68:350-386.
51. Kirk R.E. Practical significance: A concept whose time has come. Educational and Psychological Measurement 1996, 61(5):246-759.
52. Kline R.B. Beyond significance testing: Reforming data analysis methods in behavioral research 2004, American Psychological Association, Washington, DC.
53. Krantz D. The null hypothesis testing controversy in Psychology. Journal of the American Statistical Association 1999, 94(448):1372-1381.
54. Kruskal W.H. Tests of significance. International encyclopedia of statistics 1978, 944-958. Free Press, New York. W.H. Kruskal, J.M. Tanur (Eds.).
55. Larivière V., Gingras Y. Averages of ratios vs. ratios of averages: An empirical analysis of four levels of aggregation. Journal of Informetrics 2011, 5(3):392-399.
56. Leydesdorff L., Bornmann L., Mutz R., Opthof T. Turning the tables on citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology 2011, 62(7):1370-1381.
57. Leydesdorff L., Opthof T. Normalization at the field level: Fractional counting of citations. Journal of Informetrics 2010, 4(4):644-646.
58. Leydesdorff L., Opthof T. Normalization, CWTS indicators, and the Leiden Rankings: Differences in citation behavior at the level of fields 2010, http://arxiv.org/ftp/arxiv/papers/1003/1003.3977.pdf. arxiv:1003.3977v3.
59. Long R., Crawford A., White M., Davis K. Determinants of faculty research productivity in information systems: An empirical analysis of the impact of academic origin and academic affiliation. Scientometrics 2009, 78(2):231-260.
60. Lundberg J. Lifting the crown - citation z-score. Journal of Informetrics 2007, 1(2):145-154.
61. Lunneborg C.E. Data analysis by resampling 2000, Brooks Cole, Pacific Grove, CA.
62. Lykken D.T. Statistical significance in psychological research. Psychological Bulletin 1968, 70(3, part 1):151-159.
63. Marsh H.W., Jayasinghe U.W., Bond N.W. Gender differences in peer reviews of grant applications: A substantive-methodological synergy in support of the null hypothesis model. Journal of Informetrics 2011, 5(1):167-180.
64. Mayo D. Philosophy of statistics. The philosophy of science: An encyclopedia 2006, 802-815. Routledge, London. S. Sarkar, J. Pfeifer (Eds.).
65. McCloskey D.N. The loss function has been mislaid: The rhetoric of significance tests. American Economic Review 1985, 75(2):201-205.
66. McCloskey D.N., Ziliak S.T. The standard error of regression. Journal of Economic Literature 1996, 34(3):97-114.
67. Meehl P.E. Theoretical risks and tabular asterisk: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Counseling and Clinical Psychology 1978, 46:806-834.
68. Meehl P.E. Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry 1990, 1:108-141.
69. Mingers J., Xu F. The drivers of citations in management science journals. European Journal of Operational Research 2010, 205(2):422-430.
70. Moed H.F. The impact-factors debate: The ISI's uses and limits. Towards a critical, informative, accurate and policy-relevant bibliometrics. Nature 2002, 415(6873):731-732.
71. Moed H.F. CWTS crown indicator measures citation impact of a research group's publication oeuvre. Journal of Informetrics 2010, 4(3):436-438.
72. Morrison D.E., Henkel R.E. Significance tests reconsidered. American Sociologist 1969, 4:131-140.
73. Nickerson R.S. Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods 2000, 5(2):241-301.
74. Oakes M. Statistical inference: A commentary for the social and behavioral sciences 1986, Wiley, New York.
75. Opthof T., Leydesdorff L. Caveats for the journal and field normalizations in the CWTS (Leiden) evaluations of research performance. Journal of Informetrics 2010, 4(3):423-430.
76. Redner S. Citation statistics from 110 years of Physical Review. Physics Today 2005, 58(6):49-54.
77. Rosenthal R. Parametric measures of effect size. The handbook of research synthesis 1994, 231-244. Sage, New York. H. Cooper, L.V. Hedges (Eds.).
78. Rosnow R.L., Rosenthal R. Statistical procedures and the justification of knowledge in psychological science. American Psychologist 1989, 44(10):1276-1284.
79. Rothman K.J. Significance questing. Annals of Internal Medicine 1986, 105(3):445-447.
80. Royall R.M. Statistical evidence: A likelihood paradigm 1997, Chapman & Hall, London.
81. Rozeboom W.W. The fallacy of the null hypothesis significance test. Psychological Bulletin 1960, 57(5):416-428.
82. Rozeboom W.W. Good science is abductive, not hypothetico-deductive. What if there were no significance tests? 1997, 335-392. Erlbaum, Hillsdale, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
83. Scarr S. Rules of evidence: A larger context for the statistical debate. Psychological Science 1997, 8:16-17.
84. Schmidt F.L., Hunter J.E. Eight common but false objections to the discontinuation of significance testing in the analysis of research data. What if there were no significance tests? 1997, 37-64. Erlbaum, Hillsdale, NJ. L.L. Harlow, S.A. Mulaik, J.H. Steiger (Eds.).
85. Schubert A., Glänzel W. Statistical reliability of comparisons based on the citation impact of scientific publications. Scientometrics 1983, 5(1):59-74.
86. Seth A., Carlson K.D., Hatfield D.E., Lan H.W. So What? Beyond statistical significance to substantive significance in strategy research. Research methodology in strategy and management 2009, vol. 5:3-28. Elsevier JAI, New York.
87. Shaver J.P. What statistical significance testing is, and what it is not. Journal of Experimental Education 1993, 61(4):293-316.
88. Spaan J.A.E. The danger of pseudoscience in informetrics. Journal of Informetrics 2010, 4(3):439-440.
89. Stang A., Poole C., Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. European Journal of Epidemiology 2010, 25(4):225-230.
90. Stremersch S., Verniers I., Verhoef P.C. The quest for citations: Drivers of article impact. Journal of Marketing 2007, 71(3):171-193.
91. Tukey J.W. The philosophy of multiple comparisons. Statistical Science 1991, 6(1):100-116.
92. van Raan A.F.J., van Leeuwen T.N., Visser M.S., van Eck N.J., Waltman L. Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics 2010, 4(3):431-435.
93. Vieira E.S., Gomes J.A.N.F. Citations to scientific articles: Its distribution and dependence on the article features. Journal of Informetrics 2010, 4(1):1-13.
94. von Mises R. Wahrscheinlichkeit, statistik und wahrheit 1928, Springer, Wien.
95. Wainer H. One cheer for null hypothesis significance testing. Psychological Methods 1999, 4(2):212-213.
96. Waltman L., van Eck N.J., van Leeuwen T.N., Visser M.S., van Raan A.F.J. Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics 2011, 5(1):37-47.
97. Waltman L., van Eck N.J., van Leeuwen T.N., Visser M.S., van Raan A.J.F. On the correlation between bibliometric indicators and peer review: Reply to Opthof and Leydesdorff. Scientometrics 2011, 83(3):1017-1022.
98. Waltman L., van Eck N.J., van Leeuwen T.N., Visser M.S., van Raan A.F.J. Towards a new crown indicator: An empirical analysis. Scientometrics 2011, 87(3):467-481.
99. Webster E.J., Starbuck W.H. Theory building in industrial and organizational psychology. International review of industrial and organizational psychology 1988, 93-138. Wiley, London. C.L. Cooper, I. Robertson (Eds.).
100. Western B., Jackman S. Bayesian inference for comparative research. The American Political Science Review 1994, 88(2):412-423.
101. Wilkinson L. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 1999, 54(8):594-604. Task Force on Statistical Inference, APA Board of Scientific Affairs.
102. Winkler O.W. Interpreting economic and social data. A foundation of descriptive statistics 2009, Springer-Veralg, Berlin Heidelberg.