Significance testing as perverse probabilistic reasoning

BMC Medicine

Volumn 9, Issue , 2011, Pages

  Westover, M Brandon ^a   Westover, Kenneth D ^b   Bianchi, Matt T ^a  

^a MASSACHUSETTS GENERAL HOSPITAL   (United States)

^b HARVARD MEDICAL SCHOOL   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BAYESIAN LEARNING; FALSE POSITIVE RESULT; MATHEMATICAL PHENOMENA; MEDICAL INFORMATION; MEDICAL LITERATURE; PHYSICIAN; PROBABILITY; RECEIVER OPERATING CHARACTERISTIC; SENSITIVITY ANALYSIS; STATISTICAL SIGNIFICANCE; UNIVERSITY HOSPITAL; HUMAN; MEDICAL EDUCATION; QUESTIONNAIRE; STANDARD; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; EDUCATION, MEDICAL; HUMANS; PHYSICIANS; QUESTIONNAIRES;

EID: 79952047808     PISSN: None     EISSN: 17417015     Source Type: Journal    
DOI: 10.1186/1741-7015-9-20     Document Type: Article

References (130)

1. Olser W, Silverman M, Murray T, Bryan C. The Quotable Osler 2003, Philadelphia: ACP Press.
2. Horton NJ, Switzer SS. Statistical methods in the journal. The New England Journal of Medicine 2005, 353(18):1977-1979. 10.1056/NEJM200511033531823, 16267336.
3. Altman DG, Bland JM. Improving doctors' understanding of statistics. Journal of the Royal Statistical Society. Series A (Statistics in Society) 1991, 154(2):223-267.
4. Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. The Journal of the American Medical Association 2007, 298(9):1010-1022.
5. Ioannidis JPA. Why most published research findings are false. PLoS Medicine 2005, 2(8):e124. 10.1371/journal.pmed.0020124, 1182327, 16060722.
6. Friedman SB, Phillips S. What's the difference? Pediatric residents and their inaccurate concepts regarding statistics. Pediatrics 1981, 68(5):644-646.
7. Goodman SN. Toward evidence-based medical statistics. 1: the P value fallacy. Annals of Internal Medicine 1999, 130(12):995-1004.
8. Ghosh A, Ghosh K, Erwin P. Do medical students and physicians understand probability?. Quarterly Journal of Medicine 2004, 97:53-55., http://qjmed.oxfordjournals.org
9. Casscells W, Schoenberger A, Graboys TB. Interpretation by physicians of clinical laboratory results. The New England Journal of Medicine 1978, 299(18):999-1001. 10.1056/NEJM197811022991808, 692627.
10. Cahan A, Gilon D, Manor O, Paltiel O. Probabilistic reasoning and clinical decision-making: do doctors overestimate diagnostic probabilities?. Quarterly Journal of Medicine 2003, 96(10):763-769., http://qjmed.oxfordjournals.org/cgi/content/abstract/96/10/763
11. Eddy DM. Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities 1982, 249-267. Cambridge, UK: Cambridge University Press.
12. Gigerenzer G, Hoffrage U. How to improve Bayesian reasoning without instruction: frequency formats. Psychological Review 1995, 102(4):684-704., http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBkQFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Bjsessionid%3D32E0F35DFDDC7AA54764A877EBC6825A%3Fdoi%3D10.1.1.128.3201%26rep%3Drep1%26type%3Dpdf&rct=j&q=How%20to%20improve%20Bayesian%20Reasoning%20Without%20Instruction%3A%20Frequency%20formats%20no%20notes%20yet%20&ei=bJJ3TYnpEcGY8QPAyoCgDA&usg=AFQjCNHsbqpkhuNll0H7hfT4pQ6wXI9QMg&cad=rja
13. Falk R, Greenbaum CW. Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory Psychology 1995, 5:75-98.
14. Dienes Z. Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference 2008, Basingstoke: Palgrave Macmillan., http://www.loc.gov/catdir/enhancements/fy0828/2008014353-t.html
15. Gill J. The insignificance of null hypothesis significance testing. Political Research Quarterly 1999, 52(3):647-674.
16. Gigerenzer G, Murray DJ. Cognition as Intuitive Statistics 1987, Hillsdale, NJ: L. Erlbaum Associates.
17. Gigerenzer G. The superego, the ego, and the id in statistical reasoning. A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues 1993, 574. Hillsdale, NJ: L. Erlbaum Associates, Keren G, Lewis C.
18. Campbell L, Garnett W. The Life of James Clerk Maxwell. With a Selection from His Correspondence and Occasional Writings and a Sketch of His Contributions to Science 1882, London: Macmillan and Co.
19. Mumford D. The dawning of the age of stochasticity. Mathematics: Frontiers and Perspectives 2000, 197-218.
20. Oaksford M, Chater N. Bayesian Rationality: The Probabilistic Approach to Human Reasoning 2007, Oxford: Oxford University Press, [Oxford Cognitive Science Series].
21. Jaynes ET, Bretthorst GL. Probability Theory: The Logic of Science 2003, Cambridge, UK: Cambridge University Press.
22. Cox RT. The Algebra of Probable Inference 1961, Baltimore: Johns Hopkins Press.
23. Jaynes ET. How does the brain do plausible reasoning?. Maximum-Entropy and Bayesian Methods in Science and Engineering 1988, Kluwer Academic Publishers, Erickson GJ, Smith CR.
24. Horn KSV. Constructing a logic of plausible inference: a guide to Cox's theorem. International Journal of Approximate Reasoning 2003, 34:3-24.
25. Kruschke JK. Doing Bayesian Data Analysis: A Tutorial with R and BUGS 2010, Academic Press.
26. Kruschke JK. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science 2010, 1(5):658-676.
27. Fisher RA. Statistical Methods and Scientific Inference 1973, New York: Hafner Press, 3, rev. and enl.
28. Kruschke JK. What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences 2010, 14(7):293-300. 10.1016/j.tics.2010.05.001, 20542462.
29. Gelman A. Bayesian Data Analysis 2004, Boca Raton, Fla: Chapman & Hall/CRC, 2, [Texts in Statistical Science].
30. Diamond GA, Kaul S. Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials. Journal of the American College of Cardiology 2004, 43(11):1929-1939. 10.1016/j.jacc.2004.01.035, 15172393.
31. Berry DA. Bayesian clinical trials. Nature Reviews. Drug Discovery 2006, 5:27-36. 10.1038/nrd1927, 16485344.
32. Goodman SN. Introduction to Bayesian methods I: measuring the strength of evidence. Clinical Trials (London, England) 2005, 2(4):282-290. discussion 301-304, 364-378.
33. Berry DA. Introduction to Bayesian methods III: use and interpretation of Bayesian tools in design and analysis. Clinical Trials (London, England) 2005, 2(4):295-300. discussion 301-304, 364-378.
34. Louis TA. Introduction to Bayesian methods II: fundamental concepts. Clinical Trials (London, England) 2005, 2(4):291-294. discussion 301-304, 364-378.
35. Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. Bayesian methods in health technology assessment: a review. Health Technology Assessment (Winchester, England) 2000, 4(38):1-130.
36. Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. Methods in health service research. An introduction to Bayesian methods in health technology assessment. British Medical Journal (Clinical Research Ed.) 1999, 319(7208):508-512. 1116393, 10454409.
37. Jacobs RA, Kruschke JK. Bayesian learning theory applied to human cognition. Wiley Interdisciplinary Reviews: Cognitive Science 2010, 2:8-21.
38. Chater N, Manning CD. Probabilistic models of language processing and acquisition. Trends in Cognitive Sciences 2006, 10(7):335-344. 10.1016/j.tics.2006.05.006, 16784883.
39. Chater N, Oaksford M. The Probabilistic Mind: Prospects for Bayesian Cognitive Science 2008, Oxford: Oxford University Press.
40. Tenenbaum JB, Griffiths TL, Kemp C. Theory-based Bayesian models of inductive learning and reasoning. Trends in Cognitive Sciences 2006, 10(7):309-318. 10.1016/j.tics.2006.05.009, 16797219.
41. Xu F, Tenenbaum JB. Word learning as Bayesian inference. Psychological Review (New York) 2007, 114(2):245.
42. Steyvers M, Griffiths TL, Dennis S. Probabilistic inference in human semantic memory. Trends in Cognitive Sciences 2006, 10(7):327-334. 10.1016/j.tics.2006.05.005, 16793324.
43. Manning CD, Schütze H. Foundations of Statistical Natural Language Processing 1999, MIT Press.
44. Westover M, O'Sullivan J. Achievable rates for pattern recognition. Information Theory, IEEE Transactions on 2008, 54:299-320.
45. Yuille A, Kersten D. Vision as Bayesian inference: analysis by synthesis?. Trends in Cognitive Sciences 2006, 10(7):301-308. 10.1016/j.tics.2006.05.002, 16784882.
46. Grenander U, Miller M. Pattern Theory: From Representation to Inference 2007, USA: Oxford University Press.
47. Jordan MI, . (Ed) Learning in Graphical Models 1998, The MIT Press, 1, (Ed).
48. Bishop CM. Pattern Recognition and Machine Learning 2007, Springer, 1.
49. Mumford D, Desolneux A. Pattern Theory: The Stochastic Analysis of Real-World Signals (Applying Mathematics) 2010, Natick, Mass.: A K Peters.
50. MacKay DJC. Information Theory, Inference & Learning Algorithms 2002, Cambridge University Press, 1.
51. Frey BJ. Graphical Models for Machine Learning and Digital Communication 1998, The MIT Press.
52. Jelinek F. Statistical Methods for Speech Recognition 1998, The MIT Press.
53. Pearl J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference 1998, San Francisco, Calif: Morgan Kaufmann, Rev. 2nd printing, [The Morgan Kaufmann Series in Representation and Reasoning].
54. Paté-Cornell E, Guikema S. Probabilistic modeling of terrorist threats: a systems analysis approach to setting priorities among countermeasures. Military Operations Research 2002, 7(4):5-20.
55. Forrester M, Pettitt A, Gibson G. Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostatistics 2007, 8(2):383. 10.1093/biostatistics/kxl017, 16926230.
56. de Campos L, Fernández-Luna J, Huete J. Bayesian networks and information retrieval: an introduction to the special issue. Information Processing & Management 2004, 40(5):727-733.
57. Kolmogorov AN. Foundations of the Theory of Probability 1956, New York: Chelsea Pub. Co, 2d english.
58. Bayes T. An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society 1763, 53:370-418.
59. Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Kruger L. The Empire of Chance: How Probability Changed Science and Everyday Life (Ideas in Context) 1989, Cambridge [Cambridgeshire]: Cambridge University Press.
60. Krüger L. The Probabilistic Revolution 1987, Cambridge, Mass: MIT Press.
61. Yudkowsky E. An Intuitive Explanation of Bayes' Theorem. 2010, http://yudkowsky.net/rational/bayes
62. Dienes Z. Book Website for: Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference, Palgrave Macmillan. 2010, http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/inference
63. Browner WS, Newman TB. Are all significant P values created equal? The analogy between diagnostic tests and clinical research. The Journal of the American Medical Association 1987, 257(18):2459-2463.
64. Cohen J. The earth is round (p < .05). American Psychologist 1994, 49(12):997-1003., http://web.math.umt.edu/wilson/Math444/Handouts/Cohen94_earth_is_round.pdf
65. Cortina JM, Dunlap WP. On the logic and purpose of significance testing. Psychological Methods 1997, 2(2):161-172., http://www.sciencedirect.com/science/article/B6WYR-46P4P35-8/2/ac50f4ba827ae4cd48cd72896441afbc
66. Dixon P. The p-value fallacy and how to avoid it. Canadian Journal of Experimental Psychology = Revue Canadienne De Psychologie Exp'erimentale 2003, 57(3):189-202.
67. Frick RW. The appropriate use of null hypothesis testing. Psychological Methods 1996, 1(4):379-390.
68. Hagen RL. In praise of the null hypothesis statistical test. American Psychologist 1997, 52:15-24.
69. Killeen PR. An alternative to null-hypothesis significance tests. Psychological Science: A Journal of the American Psychological Society/APS 2005, 16(5):345-353. [PMID: 15869691], 1473027, 15869691.
70. Killeen ACO, jan Wagenmakers E, Grünwald P. A Bayesian perspective on hypothesis testing. Psychological Science 2006, 17.
71. Loftus GR. Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science 1996, 5(6):161-171. [ArticleType: research-article/Full publication date: Dec., 1996/Copyright © 1996 Association for Psychological Science].
72. Loftus GR. Analysis, interpretation, and visual presentation of experimental data. Stevens' Handbook of Experimental Psychology 2002, USA: John Wiley & Sons, Inc, Pashler H, Hoboken NJ., http://onlinelibrary.wiley.com/doi/10.1002/0471214426.pas0409/full
73. Nickerson RS. Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods 2000, 5(2):241-301. [PMID: 10937333], 10.1037/1082-989X.5.2.241, 10937333.
74. Trafimow D. Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem. Psychological Review 2003, 110(3):526-535. [PMID: 12885113], 10.1037/0033-295X.110.3.526, 12885113.
75. Wagenmakers EJ. A practical solution to the pervasive problem of p values. Psychonomic Bulletin & Review 2007, 14(5):779-804.
76. Wainer H. One cheer for null hypothesis significance testing. Psychological Methods 1999, 4(2):212-213.
77. Berger JO, Wolpert RL. The Likelihood Principle 1988, IMS.
78. O'Hagan A, Forster J. Kendall's advanced theory of statistics. Vol. 2B: Bayesian inference. 2004, http://adsabs.harvard.edu/abs/2004kats.book.....O
79. Royall R. Statistical Evidence: A Likelihood Paradigm 1997, Chapman & Hall/CRC.
80. Sellke T, Bayarri M, Berger J. Calibration of ρ values for testing precise null hypotheses. The American Statistician 2001, 55:62-71.
81. Stuart A, Ord J, Arnold S. Kendall's advanced theory of statistics. Vol. 2a: classical inference and the linear model. 1999,
82. Goodman SN. p values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. American Journal of Epidemiology 1993, 137(5):485-496. discussion 497-501.
83. Goodman SN. Toward evidence-based medical statistics. 2: the Bayes factor. Annals of Internal Medicine 1999, 130(12):1005-1013.
84. Wasserman L. All of Statistics: A Concise Course in Statistical Inference 2004, New York: Springer, [Springer Texts in Statistics].
85. Gill J. 'S'attaquer a l'Heritage de Fisher: Comment Tester une Hypothese en Science Sociale: Quelques Commentaires Sur Denis.' (Grappling with Fisher's legacy in social science hypothesis testing: some comments on Denis.). Journal de la Société Franc ̧aise de Statistique 2004, 145:1-9.
86. Matthews JR. Quantification and the Quest for Medical Certainty 1995, Princeton, NJ: Princeton University Press.
87. Marks HM. The Progress of Experiment: Science and Therapeutic Reform in the United States, 1900-1990 2000, Cambridge, UK: Cambridge University Press, 1st pbk, [Cambridge History of Medicine].
88. Porter TM. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life 1995, Princeton, NJ: Princeton University Press.
89. Bland JM, Altman DG. Statistics notes: Bayesians and frequentists. British Medical Journal 1998, 317:1151-1160. 1114120, 9784463.
90. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?. The Journal of the American Medical Association 1998, 280(3):280-282.
91. Clarke M, Alderson P, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals. The Journal of the American Medical Association 2002, 287(21):2799-2801.
92. Ioannidis JPA, Haidich A, Lau J. Any casualties in the clash of randomised and observational evidence? No - recent comparisons have studied selected questions, but we do need more data. British Medical Journal 2001, 322(7291):879-880. [PMC1120057], 10.1136/bmj.322.7291.879, 1120057, 11302887.
93. Lawlor DA, Smith GD, Kundu D, Bruckdorfer KR, Ebrahim S. Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence?. Lancet 2004, 363(9422):1724-1727. 10.1016/S0140-6736(04)16260-0, 15158637.
94. Vandenbroucke JP. When are observational studies as credible as randomised trials?. Lancet 2004, 363(9422):1728-1731. 10.1016/S0140-6736(04)16261-2, 15158638.
95. Michiels S, Koscielny S, Hill C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 2005, 365(9458):488-492. 10.1016/S0140-6736(05)17866-0, 15705458.
96. Ioannidis JP, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG. Replication validity of genetic association studies. Nature Genetics 2001, 29(3):306-309. 10.1038/ng749, 11600885.
97. Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. The Journal of the American Medical Association 2005, 294(2):218-228.
98. Colhoun HM, McKeigue PM, Smith GD. Problems of reporting genetic associations with complex outcomes. Lancet 2003, 361(9360):865-872. 10.1016/S0140-6736(03)12715-8, 12642066.
99. Ioannidis JPA. Genetic associations: false or true?. Trends in Molecular Medicine 2003, 9(4):135-138. 10.1016/S1471-4914(03)00030-3, 12727138.
100. Ioannidis JPA. Microarrays and molecular research: noise discovery?. Lancet 365(9458):454-455.
101. Neyman J, Pearson ES. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 1933, 231:289-337. [ArticleType: primary article/Full publication date: 1933/Copyright © 1933 The Royal Society].
102. Strunk A, Bhalla V, Clopton P, Nowak RM, McCord J, Hollander JE, Duc P, Storrow AB, Abraham WT, Wu AHB, Steg G, Perez A, Kazanegra R, Herrmann HC, Aumont MC, McCullough PA, Maisel A. Impact of the history of congestive heart failure on the utility of B-type natriuretic peptide in the emergency diagnosis of heart failure: results from the Breathing Not Properly Multinational Study. The American Journal of Medicine 2006, 119:69.e1-11., http://www.ncbi.nlm.nih.gov/pubmed/16431187
103. McCullough PA, Nowak RM, McCord J, Hollander JE, Herrmann HC, Steg PG, Duc P, Westheim A, Omland T, Knudsen CW, Storrow AB, Abraham WT, Lamba S, Wu AHB, Perez A, Clopton P, Krishnaswamy P, Kazanegra R, Maisel AS. B-type natriuretic peptide and clinical judgment in emergency diagnosis of heart failure: analysis from Breathing Not Properly (BNP) Multinational Study. Circulation 2002, 106(4):416-422. 10.1161/01.CIR.0000025242.79963.4C, 12135939.
104. Glimcher PW. Decisions, Uncertainty, and the Brain: The Science of Neuroeconomics 2003, Cambridge, Mass: MIT Press.
105. Barlow H. Redundancy reduction revisited. Network-Computation in Neural Systems 2001, 12(3):241-253.
106. Barlow H. The coding of sensory messages. Current Problems in Animal Behavior 1961, Cambridge: Cambridge University Press, Thorpe W, Zangwill O.
107. Barlow H. What is the computational goal of the neocortex?. Large Scale Neuronal Theories of the Brain 1994, 1-22. MIT Press, Koch C, Davis JL.
108. Knill DC, Richards W. Perception as Bayesian Inference 1996, Cambridge University Press.
109. Kahneman D, Slovic P, Tversky A. Judgment Under Uncertainty: Heuristics and Biases 1982, Cambridge: Cambridge University Press.
110. Elstein AS. Heuristics and biases: selected errors in clinical reasoning. Academic Medicine: Journal of the Association of American Medical Colleges 1999, 74(7):791-794.
111. Dolan JG, Bordley DR, Mushlin AI. An evaluation of clinicians' subjective prior probability estimates. Medical Decision Making 1986, 6(4):216-223. 10.1177/0272989X8600600406, 3773651.
112. Phelps MA, Levitt MA. Pretest probability estimates: a pitfall to the clinical utility of evidence-based medicine?. Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine 2004, 11(6):692-694.
113. Bornstein BH, Emler AC. Rationality in medical decision making: a review of the literature on doctors' decision-making biases. Journal of Evaluation in Clinical Practice 2001, 7(2):97-107. 10.1046/j.1365-2753.2001.00284.x, 11489035.
114. Dawson N, Arkes H. Systematic errors in medical decision making. Journal of General Internal Medicine 1987, 2(3):183-187. 10.1007/BF02596149, 3295150.
115. Laplace PS. Thérie Analytique Des Probabilités 1847, Paris: Imprimerie royale.
116. Olshausen BA, Field DJ. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381(6583):607-609. 10.1038/381607a0, 8637596.
117. Rao RPN, Olshausen BA, Lewicki MS. Probabilistic Models of the Brain: Perception and Neural Function 2002, The MIT Press.
118. Rieke F. Spikes: Exploring the Neural Code (Computational Neuroscience) 1997, Cambridge, Mass: MIT Press.
119. Koch C, Davis JL. Large-Scale Neuronal Theories of the Brain 1994, Storming Media.
120. Oaksford M, Chater N. The probabilistic approach to human reasoning. Trends in Cognitive Sciences 2001, 5(8):349-357. 10.1016/S1364-6613(00)01699-5, 11477004.
121. Baker C, Tenenbaum J, Saxe R. Bayesian models of human action understanding. Advances in Neural Information Processing Systems 2006, 18:99.
122. Griffiths TL, Tenenbaum JB. Optimal predictions in everyday cognition. Psychological Science 2006, 17(9):767-773. 10.1111/j.1467-9280.2006.01780.x, 16984293.
123. Edwards AWF. Likelihood 1992, Baltimore: Johns Hopkins University Press, Expanded.
124. Skellam JG. Models, inference, and strategy. Biometrics 1969, 25(3):457-475. 10.2307/2528899, 5824399.
125. Azizi F, Ghanbarian A, Madjid M, Rahmani M. Distribution of blood pressure and prevalence of hypertension in Tehran adult population: Tehran Lipid and Glucose Study (TLGS), 1999-2000. Journal of Human Hypertension 2002, 16(5):305-312. 10.1038/sj.jhh.1001399, 12082490.
126. Cowie MR, Struthers AD, Wood DA, Coats AJ, Thompson SG, Poole-Wilson PA, Sutton GC. Value of natriuretic peptides in assessment of patients with possible new heart failure in primary care. Lancet 1997, 350(9088):1349-1353. 10.1016/S0140-6736(97)06031-5, 9365448.
127. Leibniz GW. Dissertatio De Arte Combinatoria, in Qua Ex Arithmeticae Fundamentis Complicationum Ac Transpositionum Doctrina Novis Praeceptis Extruitur, & Usus Ambarum Per Universum Scientiarum Orbem Ostenditur; Nova Etiam Artis Meditandi, Seu Logicae Inventionis Semina Sparguntur apud Joh. Simon Fickium et Joh. Polycarp. Seuboldum, Literis Sporelianis: Lipsiae 1666,
128. Couturat L. La Logique De Leibniz Dapres Des Documents Inedits 1901, Collection historique des grands philosophes. Paris: F. Alcan.
129. Boole G. An Investigation of the Laws of Thought, on Which Are Founded the Mathematical Theories of Logic and Probabilities 1961, New York: Dover Publications.
130. Boole G. The Laws of Thought (1854) 1952, La Salle, Ill: The Open Court Pub. Co.