Consensus measurement in Delphi studies. Review and implications for future quality assurance

Technological Forecasting and Social Change

Volumn 79, Issue 8, 2012, Pages 1525-1536

  von der Gracht, Heiko A ^a  

^a EBS BUSINESS SCHOOL   (Germany)

Author keywords

Consensus; Decision; Delphi; Dissent; Expert; Stability

Indexed keywords

CONSENSUS; DECISION; DELPHI; DISSENT; EXPERT;

CONVERGENCE OF NUMERICAL METHODS; DATA PROCESSING; QUALITY ASSURANCE; STATISTICAL TESTS;

RESEARCH;

DECISION MAKING; DELPHI ANALYSIS; FORECASTING METHOD; QUALITY CONTROL;

EID: 84866023837     PISSN: 00401625     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.techfore.2012.04.013     Document Type: Article

References (114)

1. Strathern O. A Brief History of the Future. How Visionary Thinkers Changed the World and Tomorrow's Trends are 'Made' and Marketed 2007, Robinson, London.
2. Marchais-Roubelat A., Roubelat F. The Delphi method as a ritual: inquiring the Delphic Oracle. Technol. Forecast. Soc. Chang. 2011, 78:1491-1499.
3. Buchen I.H. Towards a history of futurism: the Greeks. Futures 1974, 6:429-434.
4. Dalkey N.C., Helmer O. An experimental application of the Delphi method to the use of experts. Manag. Sci. 1963, 9:458-467.
5. Rieger W.G. Directions in Delphi developments: dissertations and their quality. Technol. Forecast. Soc. Chang. 1986, 29:195-204.
6. Landeta J. Current validity of the Delphi method in social sciences. Technol. Forecast. Soc. Chang. 2006, 73:467-482.
7. Michel J.-B., Shen Y.K., Aiden A.P., Veres A., Gray M.K., The Google Books Team, Pickett J.P., Hoiberg D., Clancy D., Norvig P., Orwant J., Pinker S., Nowak M.A., Aiden E.L. Quantitative analysis of culture using millions of digitized books. Science 2011, 331:176-182.
8. Gnatzy T., Warth J., von der Gracht H., Darkow I.-L. Validating an innovative real-time Delphi approach - a methodological comparison between real-time and conventional Delphi studies. Technol. Forecast. Soc. Chang. 2011, 78:1681-1694.
9. Gordon T.J., Pease A. RT Delphi: an efficient, "round-less" almost real time Delphi method. Technol. Forecast. Soc. Chang. 2006, 73:321-333.
10. Rowe G., Wright G. Expert opinions in forecasting: the role of the Delphi technique. Principles of Forecasting: A Handbook for Researchers and Practitioners 2001, 125-144. Kluwer Academic Publishers, Boston et al. J.S. Armstrong (Ed.).
11. Fischer R.G. The Delphi method: a description. Rev. Critic. J. Acad. Librariansh. 1978, 4:64-70.
12. Strauss H.J., Zeigler L.H. Delphi, political philosophy and the future. Futures 1975, 7:184-196.
13. Häder M. Delphi-Befragungen. Ein Arbeitsbuch 2002, Westdeutscher Verlag, Wiesbaden.
14. Linstone H., Turoff M. Delphi: a brief look backward and forward. Technol. Forecast. Soc. Chang. 2011, 78:1712-1719.
15. Dunn W.N. Public Policy Analysis. An Introduction 2004, Pearson Prentice Hall, New Jersey. 3rd ed.
16. Coates J.F. An overview of futures methods. The Knowledge Base of Futures Studies 2005, Foresight International, Melbourne. Professional Edition. R.A. Slaughter (Ed.).
17. Dajani J.S., Sincoff M.Z., Talley W.K. Stability and agreement criteria for the termination of Delphi studies. Technol. Forecast. Soc. Chang. 1979, 13:83-90.
18. Chaffin W.W., Talley W.K. Individual stability in delphi studies. Technol. Forecast. Soc. Chang. 1980, 16:67-73.
19. Scheibe M., Skutsch M., Schofer J. Experiments in Delphi methodology. The Delphi Method - Techniques and Applications 1975, 262-287. Addison-Wesley, Reading. H.A. Linstone, M. Turoff (Eds.).
20. Crisp J., Pelletier D., Duffield C., Adams A., Nagy S. The Delphi method?. Nurs. Res. 1997, 46:116-118.
21. Rayens M.K., Hahn E.J. Building consensus using the policy Delphi method. Policy Polit. Nurs. Pract. 2000, 1:308-315.
22. Yang Y.N. Testing the stability of experts' opinions between successive rounds of Delphi studies. Annual Meeting of the American Educational Research Association, Chicago 2003.
23. Mitchell V.W. The Delphi technique: an exposition and application. Technol. Anal. Strateg. Manag. 1991, 3:333-358.
24. Fink A., Kosecoff J., Chassin M., Brook R.H. Consensus methods: characteristics and guidelines for use. Am. J. Public Health 1984, 74:979-983.
25. American Heritage Dictionary of the English Language 1994, Houghton-Mifflin, Boston. 3rd ed.
26. Armstrong J.S. Principles of Forecasting: A Handbook for Researchers and Practitioners 2001, Kluwer Academic Publishers, Boston et al.
27. Hall J. Decisions, decisions, decisions. Psychol. Today 1971, 5:51-54.
28. Turoff M. The design of a policy Delphi. Technol. Forecast. Soc. Chang. 1970, 2:149-171.
29. Rauch W. The decision Delphi. Technol. Forecast. Soc. Chang. 1979, 15:159-169.
30. Steinert M. A dissensus based online Delphi approach: an explorative research tool. Technol. Forecast. Soc. Chang. 2009, 76:291-300.
31. Turoff M. The Policy Delphi. The Delphi Method - Techniques and Applications 1975, 84-101. Addison-Wesley, Reading. H.A. Linstone, M. Turoff (Eds.).
32. Nowack M., Endrikat J., Guenther E. Review of Delphi-based scenario studies: quality and design considerations. Technol. Forecast. Soc. Chang. 2011, 78:1603-1615.
33. Wilkinson A. Scenario practices: in search of theory. J. Futur. Stud. 2009, 13:107-114.
34. Curry A. Acting on the future. Scenarios for Success: Turning Insights into Action 2007, 339-372. John Wiley & Sons, Chichester. B. Sharpe, K. van Der Heijden (Eds.).
35. Chermack T., Nimon K. The effects of scenario planning on participant decision-making style. Hum. Resour. Dev. Q. 2008, 19:351-372.
36. Rainer R.K., Snyder C.A., Carr H.H. Risk analysis for information technology. J. Manag. Inf. Syst. 1991, 8:129-147.
37. Tummala V.M.R., Burchett J.F. Applying a risk management process (RMP) to manage cost risk for an EHV transmission line project. Int. J. Proj. Manag. 1999, 17:223-235.
38. Schmidt R.C., Lyytinen K., Keil M., Cule P. Identifying software project risks: an international Delphi study. J. Manag. Inf. Syst. 2001, 17:5-36.
39. Addison T. E-commerce project development risks: evidence from a Delphi survey. Int. J. Inf. Manag. 2003, 23:25-40.
40. van de Linde E., van der Duin P. The Delphi method as early warning: linking global societal trends to future radicalization and terrorism in The Netherlands. Technol. Forecast. Soc. Chang. 2011, 78:1557-1564.
41. Tapio P. Disaggregative policy Delphi: using cluster analysis as a tool for systematic scenario formation. Technol. Forecast. Soc. Chang. 2003, 70:83-101.
42. Landeta J., Barrutia J. People consultation to construct the future: a Delphi application. Int. J. Forecast. 2011, 27:134-151.
43. de Meyrick J. The Delphi method and health research. Health Educ. 2003, 103:7-16.
44. Williams P.L., Webb C. The Delphi technique: a methodological discussion. J. Adv. Nurs. 1994, 19:180-186.
45. Hasson F., Keeney S. Enhancing rigour in the Delphi technique research. Technol. Forecast. Soc. Chang. 2011, 78:1695-1704.
46. Fan C.K., Cheng C.-L. A study to identify the training needs of life insurance sales representatives in Taiwan using the Delphi approach. Int. J. Train. Dev. 2006, 10:212-226.
47. Poling Mitchell M. Nursing education planning: a Delphi study. J. Nurs. Educ. 1998, 37:305-307.
48. MacCarthy B.L., Atthirawong W. Factors affecting location decisions in international operations - a Delphi study. Int. J. Oper. Prod. Manag. 2003, 23:794-818.
49. Lunsford D.A., Fussell B.C. Marketing business services in Central Europe. The challenge: a report of expert opinion. J. Serv. Mark. 1993, 7:13-21.
50. Loughlin K.G., Moore L.F. Using Delphi to achieve congruent objectives and activities in a pediatrics department. J. Med. Educ. 1979, 54:101-106.
51. Seagle E., Iverson M. Characteristics of the turfgrass industry in 2020: a Delphi study with implications for agricultural education programs. J South. Agric. Educ. Res. 2002, 52:1-13.
52. Alexandrov A.V., Pullicino P.M., Meslin E.M., Norris J.W. Agreement on disease-specific criteria for do-not-resuscitate orders in acute stroke. Stroke 1996, 27:232-237.
53. Pasukeviciute I., Roe M. The politics of oil in Lithuania: strategies after transition. Energy Policy 2000, 29:383-397.
54. Putnam J.W., Spiegel A.N., Bruininks R.H. Future directions in education and inclusion of students with disabilities: a Delphi investigation. Except. Child. 1995, 61:553-576.
55. Stewart J., O'Halloran C., Harrigan P., Spencer J.A., Barton J.R., Singleton S.J. Identifying appropriate tasks for the preregistration year: modified Delphi technique. Br. Med. J. 1999, 319:224-229.
56. Cottam H.-R., Roe M., Challacombe J. Outsourcing of trucking activities by relief organisations. J. Humanit. Assist. 2004, 1-26.
57. Islam D.M.Z., Dinwoodie J., Roe M. Promoting development through multimodal freight transport in Bangladesh. Transp. Rev. 2006, 26:571-591.
58. Chakravarti A.K., Vasanta B., Krishnan A.S.A., Dubash R.K. Modified Delphi methodology for technology forecasting case study of electronics and information technology in India. Technol. Forecast. Soc. Chang. 1998, 58:155-165.
59. Sharma D.P., Nair P.S.C., Balasubramanian R. Analytical search of problems and prospects of power sector through Delphi study: case study of Kerala State, India. Energy Policy 2003, 31:1245-1255.
60. West J.F., Cannon G.S. Essential collaborative consultation competencies for regular and special educators. J. Learn. Disabil. 1988, 21:56-63.
61. Rogers M.R., Lopez E.C. Identifying critical cross-cultural school psychology competencies. J. Sch. Psychol. 2002, 40:115-141.
62. Doke E.R., Swanson N.E. Decision variables for selecting prototyping in information systems development: a Delphi study of MIS managers. Inf. Manag. 1995, 29:173-182.
63. De Vet E., Brug J., De Nooijer J., Dijkstra A., De Vries N.K. Determinants of forward stage transitions: a Delphi study. Health Educ. Res. 2005, 20:195-205.
64. von der Gracht H., Darkow I.L. Scenarios for the logistics services industry: a Delphi-based analysis for 2025. Int. J. Prod. Econ. 2010, 127:46-59.
65. Raskin M.S. The Delphi study in field instruction revisited: expert consensus on issues and research priorities. J. Soc. Work. Educ. 1994, 30:75-89.
66. Hahn E.J., Rayens M.K. Consensus for tobacco policy among former state legislators using the policy Delphi method. Tob. Control. 1999, 8:137-140.
67. Spinelli T. The Delphi decision-making process. J. Psychol. 1983, 113:73-80.
68. Ray P.K., Sahu S. Productivity management in India: a Delphi study. Int. J. Oper. Prod. Manag. 1990, 10:25-51.
69. Zinn J., Zalokowski A., Hunter L. Identifying indicators of laboratory management performance: a multiple constituency approach. Health Care Manage. Rev. 2001, 26:40-53.
70. Buck A.J., Gross M., Hakim S., Weinblatt J. Using the Delphi process to analyze social policy implementation: a post hoc case from vocational rehabilitation. Policy Sci. 1993, 26:271-288.
71. Rowe G., Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecast. 1999, 15:353-375.
72. Rohrbaugh J. Improving the quality of group judgment: social judgment analysis and the Delphi technique. Organ. Behav. Hum. Perform. 1979, 24:73-92.
73. Erffmeyer R.C., Lane I.M. Quality and acceptance of an evaluative task: the effects of four group decision-making formats. Group Organ. Stud. 1984, 9:509-529.
74. Wechsler W. Delphi-Methode: Gestaltung und Potential für betriebliche Prognoseprozesse 1978, Verlag V Florentz GmbH, Munich.
75. Munier F., Rondé P. The role of knowledge codification in the emergence of consensus under uncertainty: empirical analysis and policy implications. Res. Policy 2001, 30:1537-1551.
76. Bardecki M.J. Participants' response to the Delphi method: an attitudinal perspective. Technol. Forecast. Soc. Chang. 1984, 25:281-292.
77. Naylor C.D., Basinski A., Baigrie R.S., Goldman B.S., Lomas J. Placing patients in the queue for coronary revascularization: evidence for practice variations from an expert panel process. Am. J. Public Health 1990, 80:1246-1252.
78. Kapoor P. A Systems Approach to Documentary Maritime Fraud 1987, Plymouth Polytechnic, Plymouth.
79. Saldanha J., Gray R. The potential for British coastal shipping in a multimodal chain. Marit. Policy Manag. 2002, 29:77-92.
80. Makukha K., Gray R. Logistics partnerships between shippers and logistics service providers: the relevance of strategy. Int. J. Logist. Res. Appl. 2004, 7:361-377.
81. Argyrous G. Statistics for Research 2005, Sage Publications, London. 2nd ed.
82. Shields T.J., Silcock G.W.H., Donegan H.A., Bell Y.A. Methodological problems associated with the use of the Delphi technique. Fire Technol. 1987, 23:175-185.
83. Riley M., Wood R.C., Clark M.A., Wilkie E., Szivas E. Researching and Writing Dissertations in Business and Management 2000, Thomson Learning, London.
84. Gordon T.J. The Delphi method. Futures Research Methodology Version 2.0 2003, American Council for the United Nations University, Washington. J.C. Glenn, T.J. Gordon (Eds.).
85. Murphy M.K., Sanderson C.F.B., Black N.A., Askham J., Lamping D.L., Marteau T., McKee C.M. Consensus development methods, and their use in clinical guideline development. Health Technol. Assess. 1998, 2:5-83.
86. Sekaran U. Research Methods for Business. A Skill Building Approach 2003, John Wiley & Sons, New York. 4th ed.
87. English G.M., Keran G.L. The prediction of air travel and aircraft technology to the year 2000 using the Delphi method. Transp. Res. 1976, 10:1-8.
88. Ludlow J. Delphi inquiries and knowledge utilization. The Delphi Method - Techniques and Applications 1975, 102-123. Addison-Wesley, Reading. H.A. Linstone, M. Turoff (Eds.).
89. Weir C.R., Hicken B.L., Rappaport H.S., Nebeker J.R. Crossing the quality chasm: the role of information technology departments. Am. J. Med. Qual. 2006, 21:382-393.
90. Ferri C.P., Prince M., Brayne C., Brodaty H., Fratiglioni L., Ganguli M., Hall K., Hasegawa K., Hendrie H., Huang Y., Jorm A., Mathers C., Menezes P.R., Rimmer E., Scazufca M. Global prevalence of dementia: a Delphi consensus study. Lancet 2005, 366:2112-2117.
91. Brender J., Ammenwerth E., Nykänen P., Talmon J. Factors influencing success and failure of health informatics systems. A pilot Delphi study. Methods Inf. Med. 2006, 45:125-136.
92. Molnar F.J., Man-Son-Hing M., Dalziel W.B., Mitchell S.L., Power B.E., Byszewski A.M., John P.S. Assessing the quality of newspaper medical advice columns for elderly readers. Can. Med. Assoc. J. 1999, 161:393-395.
93. DeLeo W. Safety educators and practitioners identify the competencies of an occupational safety and environmental health doctoral degree: an on line application of the Delphi technique. J. Saf. Health Environ. Res. 2004, 1:1-16.
94. Clark A.C., Wenig R.E. Identification of quality characteristics for technology education programs: a North Carolina case study. J. Technol. Educ. 1999, 11.
95. Brancheau J.C., Wetherbe J.C. Key issues in information systems management. MIS Q. 1987, 11:22-45.
96. Schmidt R.C. Managing Delphi surveys using nonparametric statistical techniques. Decis. Sci. 1997, 28:763-774.
97. Cooper W.W., Gallegos A., Granof M.H. A Delphi study of goals and evaluation criteria of state and privately owned latin American airlines. Socioecon. Plann. Sci. 1995, 29:273-285.
98. Hakim S., Weinblatt J. The Delphi process as a tool for decision making. The Case of Vocational Training of People with Handicaps, Evaluation and Program Planning 1993, 16:25-38.
99. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychol. Bull. 1971, 76:378-382.
100. Viera A.J., Garrett J.M. Understanding interobserver agreement: the kappa statistic. Fam. Med. 2005, 37:360-363.
101. Seigel D.G., Podgor M.J., Remaley N.A. Acceptable values of kappa for comparison of two groups. Am. J. Epidemiol. 1992, 135:571-578.
102. Fleiss J.L. Statistical Methods for Rates and Proportions 1981, John Wiley & Sons, New York. 2nd ed.
103. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
104. Müller R., Büttner P. A critical discussion of intraclass correlation coefficients. Stat. Med. 1994, 13:2465-2476.
105. Shrout P.E., Fleiss J.L. Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 1979, 86:420-428.
106. Jolson M.A., Rossow G.L. The Delphi process in marketing decision making. J. Mark. Res. 1971, 8:443-448.
107. Schoeman M.E.F., Mahajan V. Using the Delphi method to assess community health needs. Technol. Forecast. Soc. Chang. 1977, 10:203-210.
108. Hays W.L. Statistics 1994, Holt-Saunders, London. 4th ed.
109. Nelson B.W. Statistical manipulation of Delphi statements: its success and effects on convergence and stability. Technol. Forecast. Soc. Chang. 1978, 12:41-60.
110. Ecken P., Gnatzy T., von der Gracht H. Desirability bias in foresight: consequences for decision quality based on Delphi results. Technol. Forecast. Soc. Chang. 2011, 78:1654-1670.
111. Loveridge D. On Delphi questions. Ideas in Progress 2002, The University of Manchester, Manchester.
112. Helmer O. Systematic Use of Expert Opinions 1967, The RAND Corporation, Santa Monica, California.
113. Salancik J.R., Wenger W., Helfer E. The construction of Delphi event statements. Technol. Forecast. Soc. Chang. 1971, 3:65-73.
114. Linstone H.A., Turoff M. Evaluation: introduction. The Delphi Method - Techniques and Applications 1975, 229-235. Addison-Wesley, Reading. H.A. Linstone, M. Turoff (Eds.).