The validity of the meta-analytic method in addressing the issue of psi replicability

Journal of Parapsychology

Volumn 75, Issue 2, 2011, Pages 261-277

  Murray, Aja Louise ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Meta analysis; Psi; Replication; Subjectivity

Indexed keywords

EID: 84857804783     PISSN: 00223387     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (94)

1. Atmanspacher, H., & Jahn, R. G. (2003). Problems of reproducibility in complex mind-matter systems. Journal of Scientific Exploration, 17, 243-270.
2. Bailar, J. C. (1997). The promise and problems of meta-analysis. New England Journal of Medicine, 337, 559-561.
3. Bayarri, M. J., & Berger. J. (1991). Comment. Statistical Science, 6, 379-382.
4. Beloff, J. (1994, August). The sceptical position: Is it tenable? Paper presented at the 37th Annual Convention of the Parapsychological Association, Amsterdam, The Netherlands.
5. Bern, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 700, 407-425.
6. Bern, D. J., & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115, 4-18. (Pubitemid 24976733)
7. Bern, D. J., Palmer, J., & Broughton, R. S. (2001). Updating the ganzfeld database: A victim of its own success? Journal of Parapsychology, 65, 207-218. (Pubitemid 33736906)
8. Bern, D. J., Utts, J., & Johnson, W. O. (2011). Reply: Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101, 716-719.
9. Borenstein, M., Hedges, L. V., Higgin, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex: Wiley-Blackwell.
10. Bösch, H., Steinkamp, E, & Boiler, E. (2006a). Examining psychokinesis: The interaction of human intention with random number generators-A meta-analysis. Psychological Bulletin, 132, 497-523. (Pubitemid 44212066)
11. Bösch, H., Steinkamp, E, & Boiler, E. (2006b). In the eye of the beholder: Reply to Wilson and Shadish (2006) and Radin, Nelson and Houtkooper (2006). Psychological Bulletin, 132, 533-537. (Pubitemid 44212069)
12. Broughton, R. S. (1991). Parapsychology: The controversial science. New York: Ballantine.
13. Chamberlin, T. C. (1897). The method of multiple working hypotheses. Journal of Geology, 5, 837-848.
14. Cuijpers, P., Smit, E, Bohlmeijer, E., Hollon, S. D., & Andersson, G. (2010). Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: Meta-analytic study of publication bias. British Journal of Psychiatry, 196, 173-178.
15. Darlington, R. B., & Hayes, A. F. (2000). Combining independent p values: Extensions of the Stouffer and binomial methods. Psychological Methods, 5, 496-451.
16. Dawson, R. (1991). Comment. Statistical Science, 6, 382-385.
17. Delanoy, D. L. (1993, August). Experimental evidence suggestive of anomalous consciousness interactions. Proceedings of the Second Gauss Symposium, Munich, Germany.
18. Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel plot-based method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. (Pubitemid 30422636)
19. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in metaanalysis detected by a simple graphical test. British Medical Journal, 315, 629-634. (Pubitemid 27387172)
20. Eysenck, H. J. (1994). Meta-analysis and its problems. British Medical Journal, 309, 789-792. (Pubitemid 24291646)
21. Fishbain, D., Cutler, R. B., Rosomoff, H. L., & Rosomoff, R. S. (2000). What is the quality of the implemented meta-analytic procedures in chronic pain treatment meta-analyses? Clinical Journal of Pain, 16, 73-85. (Pubitemid 30155398)
22. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. London: Sage.
23. Godfrey-Smith, P. (2003). Theory and reality: An introduction to the philosophy of science. Chicago: University of Chicago Press.
24. Haraldsson, E. (1993). Are religiosity and belief in the afterlife better predictors of ESP performance than belief in psychic phenomena? Journal of Parapsychology, 57, 259-273.
25. Higgins, J. P. T., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, West Sussex: Wiley.
26. Honorton, C. (1985). Meta-analysis of psi ganzfeld research: A response to Hyman. Journal of Parapsychology, 49, 51-91.
27. Honorton, C, & Ferrari, D. C. (1989). "Future telling": A meta-analysis of forced-choice precognition experiments, 1935-1987. Journal of Parapsychology, 53, 281-308.
28. Honorton, C, Berger, R. E., Varvoglis, M. P., Quant, M., Derr, P., Schechter, E. I., & Ferrari, D. C. (1990). Psi communication in the ganzfeld: Experiments with an automated testing system and a comparison with a meta-analysis of earlier studies. Journal of Parapsychology, 54, 99-139.
29. Honorton, C, Ferrari, D. C, & Bern, D. J. (1998). Extraversion and ESP performance: A meta-analysis and a new confirmation. Journal of Parapsychology, 62, 255-276. (Pubitemid 128683299)
30. Hyman, R. (1985). The ganzfeld psi experiment: A critical appraisal. Journal of Parapsychology, 49, 3-49.
31. Hyman, R. (1991). Comment. Statistical Science, 6, 389-392.
32. Hyman, R. (2010). Meta-analysis that conceals more than it reveals: Comment on Storm et al. (2010). Psychological Bulletin, 136, 486-490.
33. Hyman, R., & Honorton, C. (1986). Ajoint communiqué: The psi ganzfeld controversy. Journal of Parapsychology, 50, 351-364.
34. Ioannidis, J. P. A. (2008). Why most true associations are inflated. Epidemiology, 19, 640-648.
35. Irwin, H. J., & Watt, C. A. (2007). An introduction to parapsychology (5th ed.). London: McFarland.
36. Jarrett, C. (2006). Evidence for psychokinesis, but is it meaningless? The psychologist. Retrieved from http://www.bps.org.uk/publications/ thepsychologist/extras/pages$/2006-news/evidence-forpsychokinesis.cfm
37. Johnson, B. T, & Eagly, A. H. (2000). Quantitative synthesis of social psychological research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 339-369). Cambridge, England: Cambridge University Press.
38. Kennedy, J. E. (2003). The capricious, actively evasive, unsustainable nature of psi: A summary and some hypotheses. Journal of Parapsychology, 67, 53-74. (Pubitemid 36803038)
39. Kennedy, J. E. (2004). A proposal and challenge for the proponents and skeptics of psi. Journal of Parapsychology, 68, 157-167. (Pubitemid 40622811)
40. Kennedy, J. E. (2006). Letter on meta-analysis in parapsychology. Journal of Parapsychology, 70, 410-413.
41. Kitcher, P. (2001). Science, truth and democracy. Oxford, England: Oxford University Press.
42. Kraemer, H. C, Gardner, C, Brooks, J. O., III, & Yesavage, J. A. (1998). Advantages of excluding underpowered studies in meta-analysis: Inclusionist vs. exclusionist viewpoints. Psychological Methods, 3, 23-31.
43. Krippner, S., Braud, W., Child, I. L., Palmer, J., Rao, R., Schlitz, M., et al. (1993). Demonstration research and meta-analysis in parapsychology. Journal of Parapsychology, 57, 277-286.
44. Laird, N. M., & Mosteller, F. (1990). Some statistical methods for combining experimental results. International Journal of Technology Assessment in Health Care, 6, 5-30. (Pubitemid 20287159)
45. Lawrence, T. R. (1993). Gathering in the sheep and goats: A meta-analysis of forced-choice studies, 1947-1993. Proceedings of Presented Papers: The Parapsychological Association 37th Annual Convention, 75-86.
46. Longino, H. (1990). Science as social knowledge: Values and objectivity in scientific enquiry. Princeton, NJ: Princeton University Press.
47. Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.
48. Macaskill, P., Walter, A. D., & Irwig, L. (2001). A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, 641-654. (Pubitemid 32166424)
49. McGraye, S. B. (2011). The theory that would not die. London: Yale University Press.
50. Milton, J. (1996). Establishing methodological guidelines for ESP studies: A questionnaire survey of experimenters' and critics' consensus. Journal of Parapsychology, 60, 289-334.
51. Milton. J. (1997a). Meta-analysis of free-response ESP studies without altered states of consciousness. Journal of Parapsychology, 61, 279-319.
52. Milton, J. (1997b). A meta-analytic comparison of the sensitivity of direct hits and sums of ranks as outcome measures for free-response studies. Journal of Parapsychology, 61, 227-241.
53. Milton, J. (1999). Should the ganzfeld research continue to be crucial in the search for a replicable psi effect? Part I. Discussion paper and introduction to an electronic mail discussion. Journal of Parapsychology, 63, 309-333.
54. Milton, J., & Wiseman, R. (1999). Does psi exist? Lack of replication of an anomalous process of information transfer. Psychological Bulletin, 125, 387-391.
55. Milton, J., & Wiseman, R. (2001). Does psi exist? A reply to Storm & Ertel (2001). Psychological Bulletin, 127, 434-438. (Pubitemid 33610291)
56. Morris, R. L. (1991). Comment. Statistical Science, 6, 393-395.
57. Munafo, M. R., & Flint, J. (2010). How reliable are scientific findings? The British Journal of Psychiatry, 197, 257-258.
58. Munafo, M. R., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor. Molecular Psychiatry, 14, 119-120.
59. Nestoriuc, Y, Kriston, L., & Rief, W. (2010). Meta-analysis as the core of evidence-based behavioral medicine: Tools and pitfalls of a statistical approach. Current Opinion in Psychiatry, 23, 145-150.
60. Noble, J. H. (2006). Meta-analysis: Methods, strengths, weaknesses and political uses. Journal of Laboratory and Clinical Medicine, 147, 7-19.
61. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157-159.
62. Palmer, J. (2003). ESP in the ganzfeld-Analysis of a debate. Journal of Consciousness Studies, 70, 51-68.
63. Palmer, J. (2010). Meta-analysis of ESP ganzfeld studies [Letter to the editor]. Skeptical Inquirer, 34, 62.
64. Radin, D. I. (1997). The conscious universe: The scientific truth of psychic phenomena. San Fransisco: HarperEdge.
65. Radin, D. I. (2005). The sense of being stared at: A preliminary metaanalysis. Journal of Consciousness Studies, 72, 95-100.
66. Radin, D. I., & Ferrari, D. C. (1991). Effects of consciousness on the fall of the dice: A meta-analysis. Journal of Scientific Exploration, 5, 61-83.
67. Radin, D. I., & Nelson, R. D. (1989). Evidence for consciousness-related anomalies in random physical systems. Foundations of Physics, 19, 1499-1514.
68. Radin, D. I., & Nelson, R. D. (2000). Meta-analysis of mind-matter interaction experiments. Unpublished manuscript, Boundary Institute, Los Altos, and Princeton Engineering Anomalies Research, Princeton University [cited in Kennedy, 2006].
69. Radin, D. I., & Nelson, R. D. (2003). Research in mind-matter interactions (MMI): Individual intention. In W. B. Jonas & C. C. Crawford (Eds.), Healing intention and energy medicine: Research and clinical implications (pp. 39-48). Edinburgh, Scodand: Churchill Livingstone.
70. Radin, D. I., Nelson, R. D., Dobyns, Y., & Houtkooper, J. (2006). Reexamining psychokinesis: Comment on Bösch, Steinkamp, and Boiler (2006) meta-analysis. Psychological Bulletin, 132, 529-532. (Pubitemid 44212068)
71. Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641.
72. Rosenthal, R. (1986). Meta-analytic procedures and the nature of replication: The ganzfeld debate. Journal of Parapsychology, 50, 315-336.
73. Rosenthal, R. (1991). Meta-analysis: A review. Psychosomatic Medicine, 53, 247-271.
74. Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, 183-192.
75. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Hoboken, NJ: Wiley.
76. Schlitz, M. J., & Braud, W. G. (1997). Distant intentionality and healing: Assessing the evidence. Alternative Therapies in Health and Medicine, 3, 62-73.
77. Schmidt, S., Schneider, R., Utts, J., & Walach, H. (2004). Distant intentionality and the feeling of being stared at. British Journal of Psychology, 95, 235-247. (Pubitemid 38739806)
78. Sherwood, S. J., & Roe, C. A. (2003). A review of dream ESP studies conducted since the Maimonides dream ESP studies. In J. Alcock, J. Burns, & A. Freeman (Eds.), Psi wars: Getting to grips with the paranormal (pp. 85-110). Thorverton, Exeter: Imprint Academic.
79. Slavin, R. E. (1995). Best-evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology, 48, 9-18.
80. Stanford, R. G., & Stein, C. (1994). A meta-analysis of ESP studies contrasting hypnosis and a comparison condition, Journal of Parapsychology, 58, 235-269.
81. Steinkamp, F. (1998). A guide to independent coding in meta-analysis. Proceedings of Presented Papers: The Parapsychological Association 41st Annual Convention, 243-259.
82. Steinkamp, R, Milton, J., & Morris, R. (1998). A meta-analysis of forcedchoice experiments comparing clairvoyance and precognition. Journal of Parapsychology, 62, 193-218. (Pubitemid 128683297)
83. Steinkamp, R, Boiler, E., & Bosch, H. (2002). Experiments examining the possibility of human intention interactions with random number generators: A preliminary meta-analysis [Abstract]. Journal of Parapsychology, 66, 238-239.
84. Storm, L. (2000). Research note. Replicable evidence of psi: A revision of Milton's (1999) meta-analysis of the ganzfeld databases. Journal of Parapsychology, 64, 411-416.
85. Storm, L. (2006). Meta-analysis in parapsychology: I. The ganzfeld domain. Australian Journal of Parapsychology, 6, 35-53.
86. Storm, L., & Ertel, S. (2001). Does psi exist? Comments on Milton and Wiseman's (1999) meta-analysis of ganzfeld research. Psychological Bulletin, 127, 424-433.
87. Storm, L., Tressoldi, P. E., & Di Riso, L. (2010). Meta-analysis of freeresponse studies, 1992-2008: Assessing the noise reduction model in parapsychology. Psychological Bulletin, 136, 471-485.
88. Utts, J. (1991). Replication and meta-analysis in parapsychology. Statistical Science, 6, 363-378.
89. Utts, J. (1993). Honorton the meta-analyst. Journal of Parapsychology, 53, 89-100.
90. Utts, J., Norris, M., Suess, E., & Johnson, W. (2010). The strength of evidence versus the power of belief: Are we all Bayseans? In C. Reading (Ed.), Data and context in statistics education: Towards an evidence-based society. Proceedings of the 8th International Conference on Teaching Statistics. Voorburg, The Netherlands: International Statistical Institute.
91. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426-432.
92. Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259-264.
93. Watt, C. (2005). Presidential address: Parapsychology's contribution to psychology: A view from the front line. Journal of Parapsychology, 69, 215-232.
94. Wilson, D. B., & Shadish, W. R. (2006). On blowing trumpets to the tulips: To prove or not to prove the null hypothesis-Comment on Bosch, Steinkamp, and Boiler (2006). Psychological Bulletin, 132, 524-528. (Pubitemid 44212067)