Evaluation of nine consensus indices in delphi foresight research and their dependency on delphi survey characteristics: A simulation study and debate on delphi design and interpretation

PLoS ONE

Volumn 10, Issue 8, 2015, Pages

  Birko, Stanislav ^a   Dove, Edward S ^b   Özdemir, Vural ^c,d   Dalal, Koustuv ^e  

^a MCGILL UNIVERSITY   (Canada)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c GAZIANTEP UNIVERSITY   (Turkey)

^d AMRITA UNIVERSITY   (India)

^e ÖREBRO UNIVERSITY   (Sweden)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CALCULATION; DELPHI STUDY; MATHEMATICAL COMPUTING; RESEARCH; SAMPLE SIZE; SIMULATION; STUDY DESIGN; COMPUTER SIMULATION; HUMAN; MEDICAL RESEARCH; STANDARDS; STATISTICAL ANALYSIS;

BIOMEDICAL RESEARCH; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; DELPHI TECHNIQUE; HUMANS; SAMPLE SIZE;

EID: 84943190932     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0135162     Document Type: Article

References (39)

1. Twycross A (2001) Achieving consensus about pain content for child branch curricula. J Adv Nursing 34(1): 51-60.
2. Fink A, Kosecoff J, Chassin M, Brook RH (1983) Consensus methods: Characteristics and guidelines for use. Am J Pub Health 74: 979-983.
3. Keeney S, Hasson F, McKenna H (2011) The Delphi Technique in Nursing and Health Research. Oxford: Wiley-Blackwell.
4. Delbecq AL, Van de Ven A, Gustafson D (1975) Group Techniques for Program Planning: A Guide to Normal Group and Delphi Processes. Glenview, Illinois: Scott, Foreman and Company.
5. Sinha IP, Smyth RL, Williamson P.R. (2011) Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 8(1): e1000393. doi: 10.1371/journal.pmed.1000393 PMID: 21283604
6. Birko S, Dove ES, Özdemir V (2015) A delphi technology foresight study: Mapping social construction of scientific evidence on metagenomics tests for water safety. PLoSOne 10(6): e0129706. doi: 10. 1371/journal.pone.0129706 PMID: 26066837
7. Adelson M, Aroni S. (1975) Differential images of the future. In: Linstone H, Turoff M, editors. The Delphi Method: Techniques and Applications. Reading: Addison-Wesley. pp. 433-462.
8. Meijering JV, Kampen JK, Tobi H. (2013) Quantifying the development of agreement among experts in Delphi studies. Technol Forecast Soc Change 80:1607-1614.
9. Warrens MJ (2010) Inequalities between multi-rater kappas. Adv Data Anal Classif 4: 271-286.
10. Uebersax JS (1987) Diversity of decision-making models and the measurement of interrater agreement. Psychol Bull 101:140-146.
11. Fleiss JL (1975) Measuring agreement between two judges on the presence or absence of trait. Biometrics 31: 651-659. PMID: 1174623
12. Conger AJ (1971) Integration and generalization of kappas for multiple raters. Psychol Bull 76: 378-382.
13. von der Gracht HA (2012) Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technol Forecast Soc Change 79:1525-1536.
14. Keeney S, Hasson F, McKenna HP (2006) Consulting the oracle: Ten lessons from using the Delphi technique in nursing research. J Adv Nursing 53(2): 205-212.
15. Nowack M, Endrikat J, Guenther E (2011) Review of Delphi-based scenario studies: Quality and design considerations. Technol Forecast Soc Change 78(9): 1603-1615.
16. Fitch K, Bernstein SJ, Aguilar M, Burnand B, LaCalle J, Lazaro P, et al (2001) RAND/UCLA Appropriateness Method (RAM). Santa Monica: RAND Corporation.
17. Di Mare A., Latora V (2007) Opinion formation models based on game theory. International J Modern PhysicsC 18(9): 1377-1395.
18. Proctor S, Hunt M. (1994) Using the Delphi survey technique to develop a professional definition of nursing for analysing nursing workload. J Adv Nursing 19:1003-1014.
19. Beech BF (1997) Studying the future: a Delphi survey of how multidisciplinary clinical staffs view the likely development of two community mental health centres over the course of the next two years. J Adv Nursing 25:331-338.
20. Green B, Jones M, Hughes D., Williams A. (1999) Applying the Delphi technique in a study of GPs information requirement. Health Soc Care Comm 7(3): 198-205.
21. Özdemir V, Knoppers BM (2014) From government to anticipatory governance: responding to the challenges of innovation and emerging technologies. In: Smart governance for health and well-being: the evidence. Kickbusch I, Gleicher D. Editors. Copenhagen: World Health Organization, pp. 12-33.
22. French P, Yin-Yu H, Lan-Suen L (2002) A Delphi survey of evidence-based nursing priorities in Hong Kong. Journal of Nursing Management 10: 265-273. PMID: 12191072
23. Özdemir V, Kiliç H, Yildirim A, Vayena E, Dove ES, Gungor K, et al (2015) A code of ethics for ethicists: What would Pierre Bourdieu say? "Do not misuse social capital in the age of consortia ethics". Am J Bioethics 15: 64-67.
24. Hatchard JL, Fooks GJ, Evans-Reeves KA, Ulucanlar S, Gilmore AB (2014) A critical evaluation of the volume, relevance and quality of evidence submitted by the tobacco industry to oppose standardised packaging of tobacco products. BMJ Open 4(2): e003757. doi: 10.1136/bmjopen-2013-003757 PMID: 24523419
25. Özdemir V, Kolker E, Hotez PJ, Mohin S, Prainsack B, Wynne B, et al (2014) Ready to put metadata on the post-2015 development agenda? Linking data publications to responsible innovation and science diplomacy. OMICS 2014; 18(1): 1-9.
26. Dandara C, Huzair F, Borda-Rodriguez A, Chirikure S, Okpechi I, et al (2014) H3Africa and the African Life Sciences Ecosystem: Building Sustainable Innovation. OMICS 18(12): 733-739. doi: 10.1089/omi.2014.0145 PMID: 25454511
27. Özdemir V, Endrenyi L, Aynacioʇlu S, Bragazzi NL, Dandara C, Dove ES, et al (2014) Bernard Lerer: recipient of the 2014 inaugural Werner Kalow Responsible Innovation Prize in Global Omics and Personalized Medicine (Pacific Rim Association for Clinical Pharmacogenetics). OMICS 18(4): 211-221. doi: 10.1089/omi.2014.0029 PMID: 24649998
28. Hekim N, Coşkun Y, Snav A, Abou-Zeid AH, Aʇirbash M, Akintola SO, et al (2014) Translating biotechnology to knowledge-based innovation, peace, and development? Deploy a Science Peace Corps - an open letter to world leaders. OMICS 18(7): 415-420. doi: 10.1089/omi.2014.0079 PMID: 24955641
29. Ulucanlar S, Faulkner A, Peirce S, Elwyn G (2013) Technology identity: the role of sociotechnical representations in the adoption of medical devices. Soc Sci Med 98: 95-105. doi: 10.1016/j.socscimed.2013.09.008 PMID: 24331887
30. Birch K, Tyfield D. (2013) Theorizing the Bioeconomy: Biovalue, Biocapital, Bioeconomics or⋯ What? Science Technol Human Values 38: 299-327.
31. Dove ES, Ozdemir V. (2013) 'Regular science' is inherently political. EMBO Rep 14(2), 113.
32. Thoreau F, Delvenne P. (2012). Have STS fallen into a political void? Depoliticisation and engagement in the case of nanotechnologies. Politica Sociedade 11: 205-226.
33. Berkson J (1938) Some difficulties of interpretation encountered in the application of the chi-square test. J Am Stat Ass 33: 526-542.
34. White JW, Rassweiler A, Samhouri J.F., Stier AC, White C. (2014). Ecologists should not use statistical significance tests to interpret simulation model results. Oikos 123: 385-389.
35. Gwet K (2002) Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Stat Methods Inter-Rater Reliability Assessment 1:1-5.
36. Feinstein AR, Cicchetti DV(1990) High agreement but low Kappa: I. The problems of two paradoxes. J Clin Epidemiol 43: 543-549. PMID: 2348207
37. Wonkam A, Hurst S (2014) A call for policy action in sub-Saharan Africa to rethink diagnostics for pregnancy affected by sickle cell disease: differential views of medical doctors, parents and adult patients predict value conflicts in Cameroon. OMICS 18(7): 472-480. doi: 10.1089/omi.2013.0167 PMID: 24754796
38. Güngör K, Hotez PJ, Özdemir V, Aynacioʇlu Ş (2014) Glaucomics: a call for systems diagnostics for 21 (st) century ophthalmology and personalized visual health. OMICS 18(5): 275-279. doi: 10.1089/omi.2014.0040 PMID: 24730382
39. Faucheux S, Hue C. (2001) From irreversibility to participation: towards a participatory foresight for the governance of collective environmental risks. J Hazard Mater 86(1-3): 223-243. PMID: 11532368