Enhancing rigour in the Delphi technique research

Technological Forecasting and Social Change

Volumn 78, Issue 9, 2011, Pages 1695-1704

  Hasson, Felicity ^a   Keeney, Sinead ^a  

^a UNIVERSITY OF ULSTER   (United Kingdom)

Author keywords

Delphi technique; Reliability; Trustworthiness; Validity

Indexed keywords

DELPHI STUDY; DELPHI TECHNIQUE; GOOD RESEARCH; QUANTITATIVE MEASUREMENT; TRUSTWORTHINESS; VALIDITY;

INNOVATION; RELIABILITY; TECHNOLOGICAL FORECASTING;

FORECASTING;

DELPHI ANALYSIS; EXPERIMENTAL STUDY; PLANNING METHOD; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; RELIABILITY ANALYSIS; RESEARCH WORK;

EID: 82455167992     PISSN: 00401625     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.techfore.2011.04.005     Document Type: Article

References (138)

1. Long T., Johnson M. Rigour, reliability and validity in qualitative research. Clin. Effect. Nurs. 2000, 4:30-37.
2. Lacey A. The research process. The Research Process in Nursing 2010, 13-35. Wiley-Blackwell, United Kingdom.
3. Kirk J., Miller M.L. Reliability and Validity in Qualitative Research 1986, Sage Publications, Beverly Hills.
4. Waltz C.F., Stricjland O.L., Lenz E.R. Measurement in Nursing and Health Research 2005, Springer, New York. 3rd ed.
5. Patton M. Qualitative Research and Evaluation Methods 2002, Sage Publications, Thousand Oaks, CA. 3rd ed.
6. Mason E.J., Bramble W.J. Understanding and Conducting Research 1989, McGraw-Hill, New York. 2nd ed.
7. McIntire S.A., Miller L.A. Foundations of Psychological Testing 2005, Sage Publishing Company, London and New York.
8. Krefting L. Rigour in qualitative research: the assessment of trustworthiness. Am. J. Occup. Ther. 1991, 45(3):214-222.
9. Lincoln Y.S., Guba E.G. Naturalistic Inquiry 1985, Sage Publications Inc, Newbury Park, London.
10. Polit D., Beck C., Hungler B. Essentials of Nursing Research - Methods, Appraisal and Utilisation 2001, Lippincott, Philadelphia.
11. Cornick P. Nitric oxide education survey - use of a Delphi survey to produce guidelines for training neonatal nurses to work with inhaled nitric oxide. J. Neonatal Nurs. 2006, 12(2):62-68.
12. Golafshani N. Understanding reliability and validity in qualitative research. Qual. Rep. 2003, 8(4):597-607.
13. Dalkey N.C., Helmer O. An experimental application of the Delphi method to the use of experts. Manage. Sci. 1963, 9(3):458-467.
14. Rauch W. The decision Delphi. Technol. Forecast. Soc. Change 1979, 15(3):159-169.
15. McKenna H.P. The Delphi technique: a worthwhile approach for nursing?. J. Adv. Nurs. 1994, 19:1221-1225.
16. Crisp J., Pelletier D., Duffield C., Adams A., Nagy S. The Delphi method?. Nurs. Res. 1997, 46(2):116-118.
17. Beretta R. A critical review of the Delphi technique. Nurse Res. 1996, 3:79-89.
18. Turoff M. Personal communication, 2006. Environ. Plann. A 2008, 40:1485-1500. N. Novakowski, B. Wellar (Eds.).
19. The Delphi method: techniques and applications 1975, 262-287. Addison-Wesley, Massachusetts. H.A. Linstone, M. Turoff (Eds.).
20. Reid N.G. The Delphi technique: its contribution to the evaluation of professional practice. Professional Competence and Quality Assurance in the Caring Profession 1988, 230-262. Croom Held, New York. E. Ellis (Ed.).
21. Rowe G., Frewer L.J. A typology of public engagement mechanisms. Sci. Technol. Hum. Val. 2005, 30(2):251-290.
22. Woudenberg F. An evaluation of Delphi. Technol. Forecast. Soc. Change 1991, 40:131-150.
23. Rowe G., Wright G., Bolger F. Delphi: a re-evaluation of research and theory. Technol. Forecast. Soc. Change 1991, 39:235-251.
24. Linstone H. The Delphi technique. Handbook of Futures Research 1978, Greenwood Press, Westport. J. Fowles (Ed.).
25. Mays N., Pope C.P. Quality in qualitative health research. Qualitative Research in Health Care 2000, 89-101. BMJ Books, London. C.P. Pope, N. Mays (Eds.).
26. Altheide D., Johnson J. Criteria for assessing interpretive validity in qualitative research. Handbook of Qualitative Research 1994, 485-499. Sage Publications, London. N.K. Denzin, Y.S. Lincoln (Eds.).
27. Guba E.G. Criteria for assessing the trustworthiness of naturalistic inquiries. Educ. Comm. Tech. J. 1981, 29:75-91.
28. Coates J.F. In defence of Delphi. Technol. Forecas. Soc. Change 1975, 7:193-194.
29. Linstone H.A. Eight basic pitfalls: a checklist. The Delphi Method: Techniques and Applications 1975, 573-586. Addison-Wesley, Reading MA. H.D. Linstone, M. Turoff (Eds.).
30. Fink A., Kosecoff J., Chassin M., Brook R. Consensus Methods: Characteristics and Guidelines for Use 1991, RAND, Santa Monica, CA.
31. Powell C. The Delphi technique: myths and realities. J. Adv. Nurs. 2003, 41(4):376-382.
32. Ament R.H. Comparison of Delphi forecasting studies in 1964 and 1969. Futures 1970, 1:35-44.
33. Uhl N.P. Consensus and the Delphi process. Paper Presented at the Annual Meeting of the American Educational Research Association (Washington, DC, April) 1975, (ERIC Document ED 104201.
34. Duffield C. The Delphi technique: a comparison of results obtained using two expert panels. Int. J. Nurs. Stud. 1993, 30(3):227-237.
35. Holloway I., Wheeler S. Qualitative Research for Nurses 1996, Blackwell Science, Oxford.
36. Day J., Bobeva M. A generic toolkit for the successful management of Delphi studies. EJBRM 2005, 3(2):103-116.
37. Lindeman C. Delphi survey of priorities in clinical nursing research. Nurs. Res. 1975, 24:434-441.
38. Jillson I. Developing guidelines for the Delphi method. Technol. Forecas. Soc. Change 1975, 7:221-222.
39. Van Zolingen S.J., Klaassen C.A. Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education. Technol. Forecast. Soc. Change 2003, 70(4):317-340.
40. Hardy D.J., O'Brien A.P., Gaskin C.J., O'Brien A.J., Morrison-Ngatal E., Skews G., Ryan T., McNulty N. Practical application of the Delphi technique in a bicultural mental health nursing study in New Zealand. J. Adv. Nurs. 2004, 46(1):95-109.
41. Keeney S. The Delphi technique. The Research Process in Nursing 2009, Blackwell Publishing, London. K. Gerrish, A. Lacey (Eds.).
42. McKenna H.P., Keeney S. Delphi studies. Nursing Research: Designs and Methods 2008, 251-260. Churchill Livingstone, Edinburgh. R. Watson, H.P. McKenna, S. Cowman, J. Keady (Eds.).
43. Turoff M. The design of a policy Delphi. J. Technol. Forecast. Soc. Change 1970, 2(2):149-171.
44. Turoff M. Delphi conferencing: computer-based conferencing with anonymity. Technol. Forecast. Soc. Change 1972, 3:159-204.
45. Kuusi O. Expertise in the future use of generic technologies - epistemic and methodological considerations concerning Delphi studies 1999, VATT-Research Report, Government Institute for Economic Research, Helsinki.
46. Tapio P. Disaggregative policy Delphi: using cluster analysis as a tool for systematic scenario formation. Technol. Forecast. Soc. Change 2003, 70(1):83-101.
47. Kahneman D., Slovic P., Tversky A. Judgment under Uncertainty: Heuristics and Biases 1982, Cambridge University Press, Cambridge.
48. Kastein M.R., Jacobs M., Van Der Hell R.H., Luttik K., Touw-Otten F.W.M.M. Delphi, the issue of reliability: a qualitative Delphi study in primary health care in the Netherlands. Technol. Forecast. Soc. Change 1993, 44:315-323.
49. Gordon T.J. The methods of futures research (electronic version). Ann. Amer. Acad. Polit. Soc. Sci. 1992, 522:25-36.
50. Ziglio E. The Delphi method and its contribution to decision-making. Gazing into the Oracle: the Delphi Method and Its Application to Social Policy and Public Health 1996, 3-33. Jessica Kingsley Publishers, London. M. Adler, E. Ziglio (Eds.).
51. Clayton M.J. Delphi: a technique to harness expert opinion for critical decision-making tasks in education (electronic version). Educational Psychology: An International Journal of Experimental Educational Psychology 1997, 17:373-387.
52. Sackman H. Delphi-critique: expert opinion. Forecasting and Group Process 1975, Lexington Books, Massachusetts.
53. Williams P., Webb C. The Delphi technique: a methodological discussion. J. Adv. Nurs. 1994, 19:180-186.
54. Yousuf M.I. Using experts' opinions through Delphi technique. PARE 2007, 12(4). accessed 02-09.2010. http://pareonline.net/pdf/v12n4.pdf.
55. Morgan P.J., Lam-McCullough J., Herold-McIlroy J., Tarshis J. Simulation performance checklist generation using the Delphi technique. Can. J. Anesth. 2007, 52(12):992-997.
56. Mullen P.M. Delphi: myths and reality. J. Health Organ. Manag. 2003, 17(1):37-52.
57. Loo R. The Delphi method: a powerful tool for strategic management. Policing 2002, 25(4):762-769.
58. Quintana J.M., Arostegui I., Azkarate J., Goenaga J.I., Elexpa X., Letona J., Arcelay A. Evaluation of explicit criteria for total hip joint replacement. J. Clin. Epidemiol. 2000, 53(12):1200-1208.
59. Berra S., Sanchez E., Pons J.M.V., Tebe C., Alonso J., Aymerich M. Setting priorities in clinical and health services research: properties of an adapted and updated method. Int. J. Technol. Assess. Health Care 2010, 26(2):217-224.
60. Samarakkody D.C., Fernando D.N., Perera H., McClure R.J., De Silva H. The Child Behaviour Assessment Instrument: development and validation of a measure to screen for externalising child behavioural problems in community setting. Int. J. Ment. Health Syst. 2010, 4(13). accessed 13-09.2010. http://www.ijmhs.com/content/4/1/13.
61. Teisl M.F., Boyle K.J., McCollum D., Reiling S.D. Test-retest reliability of contingent valuation with independent sample pretest and posttest control groups. Am. J. Agric. Econ. 1995, 77(3):613-619.
62. Helmer O. The Delphi method: an illustration. Technological Forecasting for Industry and Government 1968, 117-122. Prentice-Hall, Englewood Cliffs NJ. J.R. Bright (Ed.).
63. Okoli C., Pawlowski S.D. The Delphi method as a research tool: an example, design considerations and applications. Inform. Manage. 2004, 42:15-29.
64. Stevenson J.S. Development of nursing knowledge: accelerating the pace. The Nursing Profession Turning Points 1990, 597-606. Mosby, St Louis. N.L. Chaska (Ed.).
65. Maceviciute E., Wilson T.D. A Delphi investigation into the research needs in Swedish librarianship. Information Research 2009, 14(4). 06-09.2010. http://informationr.net/ir/14-4/paper419.html.
66. Thompson M. Considering the implication of variations within Delphi research. Fam. Pract. 2009, 26:420-424.
67. Welty G. A critique of the Delphi technique. Proceedings of American Statistical Association 1971, 377-382.
68. Welty G. Problems of selecting experts for Delphi exercises. Acad. Manage. J. 1972, 15:121-124.
69. Polit D., Beck C. Nursing Research: Principles and Methods 2004, Lippincott Williams & Wilkins, Philadelphia.
70. Brink P.J., Wood M.J. Advanced Design in Nursing Research 1998, Sage, Thousand Oaks, CA.
71. Brennan R. An essay on the history and future of reliability from the perspective of replications. J. Educ. Meas. 2001, 38(4):295-317.
72. DeVon H.A., Block M., Moyle-Wright P., Ernst D., Hayden S., Lazzara D., Savoy S., Kostas-Polston E. A psychometric toolbox for testing validity and reliability. J. Nurs. Scholarsh. 2007, 39(2):155-164.
73. Gordon T.J. The Delphi method. Futures Research Methodology, AC/UNC Millennium Project Available online:, [accessed 16th August 2010]. http://www.gerenciamento.ufba.br/Downloads/delphi%20(1).pdf.
74. Murry J.W., Hammons J.O. Delphi: a versatile methodology for conducting qualitative research. Rev. High. Educ. 1995, 18:423-436.
75. De Meyrick J. The Delphi method and health research. Health Educ. 2002, 103(1):7-16.
76. Goodman C.M. The Delphi technique: a critique. J. Adv. Nurs. 1987, 12:729-734.
77. Caves R. Consultative methods for extracting expert knowledge about clinical competence. Professional Competence and Quality Assurance in the Caring Professions 1988, 199-299. Chapman Hall, London. Ellis (Ed.).
78. Walker A.M., Self J. The Delphi method: a useful look for the allied health researcher. British J. Ther. Rehabil. 1996, 3(12):677-681.
79. Sharkey S.B., Sharples A.Y. An approach to consensus building using the Delphi technique: developing a learning resource in mental health. Nurs. Educ. Today 2001, 21:398-408.
80. Huang H.C., Lin W.C., Lin J.D. Development of a fall-risk checklist using the Delphi technique. J. Clin. Nurs. 2008, 17(17):2275-2283.
81. Spencer-Cooke B. Conditions of participation in rural, non-formal education programs: a Delphi study. Educ. Media Int. 1989, 26(2):115-124.
82. Cross V. The same but different: a Delphi study of clinicians' and academics' perceptions of physiotherapy undergraduates. Physiotherapy 1999, 85(1):28-39.
83. Marchant E.W. Methodological problems associated with the use of the Delphi technique: some comments. Fire Technol. 1988, 24(1):59-62.
84. McCampbell W.H., Stewart B.R. Career ladder programs for vocational education: desirable characteristics. J. Voc. Educ. Res. 1992, 17(1):53-68.
85. Hsu C.-C., Sandford B.A. Minimizing non-response in the Delphi process: how to respond to non-response. PARE 2007, 12(17). Available online:, [accessed 3rd March 2010]. http://pareonline.net/pdf/v12n17.pdf.
86. Schmidt R.C. Managing Delphi surveys using non-parametric statistical techniques. Decision Sci. 1997, 28(3):763-774.
87. Hasson F., Keeney S., McKenna H.P. Research guidelines for the Delphi survey technique. J. Adv. Nurs. 2000, 32:1008-1015.
88. von der Gracht H.A. The Future of Logistics: Scenarios for 2025 2008, Wissenschaft, Germany. Gabler Edition.
89. Keeney S., Hasson F., McKenna H.P. A critical review of the Delphi technique as a research methodology for nursing. Int. J. Nurs. Stud. 2001, 38:195-200.
90. Dalkey N.C. The Delphi Method: An Experimental Study of Group Opinion, Document Number - RM-5888-PR 1969, The Rand Corporation, Santa Monica, California.
91. Dalkey N.C., Brown B., Cochran S. The Delphi Method III: Use of Self-Ratings to Improve Group Estimates, Document Number - RM-6115-PR 1969, The Rand Corporation, Santa Monica, California.
92. Dalkey N.C., Brown B., Cochran S. The Delphi Method IV: Effects of Percentile Feedback and Feed-in of Relevant Facts, Document Number - RM-6118-PR 1970, The Rand Corporation, Santa Monica, California.
93. Jolson M.A., Rossow G.L. The Delphi process in marking decision making. J. Mark. Res. 1971, VIII:443-448.
94. Parente R.J., Hiob T.N., Sliver R.A., Jenkins C., Poe M., Mullins J.R. The Delphi method, impeachment and terrorism: accuracies of short-range forecasts for volatile world events. Technol. Forecas. Soc. Change 2005, 72(4):401-411.
95. Riggs W.E. The Delphi technique: an experimental evaluation. Technol. Forecast. Soc. Change 1983, 23:89-94.
96. Cicarelli J. The future of economics: a Delphi study. Technol. Forecast. Soc. Change 1984, 25(2):139-157.
97. Ono R., Wedemeyer D.J. Assessing the validity of the Delphi Technique. Futures 1994, 26(3):289-304.
98. Helmer O. The Systematic Use of Expert Judgement in Operations Research 1963, The Rand Corporation, Santa Monica, California.
99. Helmer O. Convergence of Expert Consensus through Feedback 1964, The Rand Corporation, Santa Monica, California.
100. Brown B., Helmer O. Improving the Reliability of Estimates Obtained from Consensus of Experts. Document No- p2986 1964, The RAND Corporation, Santa Monica, California.
101. Rescher N. Delphi and values. Document Number - P-4182 1969, The Rand Corporation, Santa Monica California.
102. Dalkey N.C. An experimental study of group opinion: the Delphi method. Futures 1969, 1(5):408-426.
103. Dalkey N.C. Analysis from a group opinion. Futures 1969, 1(6):541-551.
104. Dalkey N.C. Comparison of Group Judgement Techniques with Short-Range Predictions and Almanac Questions, Document Number R-678 1971, The Rand Corporation, New York.
105. Dalkey N.C. Toward a theory of group estimation. The Delphi Method: Techniques and Applications 1975, 236-261. Addison-Wesley, Reading, MA. H.L. Linstone, M. Turoff (Eds.).
106. A.W. Dahl, Delphic and interactive committee processes in a comprehensive health planning advisory council: a comparative case study. Unpublished PhD Thesis, Baltimore, MD: Johns Hopkins University School of Hygiene and Public Health, 1974.
107. J. Sack, A test of the applicability of the Delphi method of forecasting as an aid to planning in a commercial banking institution. Unpublished D.B.A. dissertation. Arizona State University, Arizona, 1974.
108. Scheive M., Skutsch M., Schofer J. Experiments in Delphi methodology. The Delphi Method: Techniques and Applications 1975, 262-287. Addison-Wesley, Massachusetts. H.A. Linstone, M. Turoff (Eds.).
109. G. Penfield, The relative efficacy of varying applications of face to face interaction versus Delphi in developing consensus about relative priority among goals in student affairs. Unpublished Dissertation, University of Cincinnati, 1975. In: W. Riggs (Ed.), The Delphi technique: an experimental evaluation, Technological Forecasting and Social Change 23 (1983), 89-94.
110. Van de Ven A.H., Delbecq A.L. The effectiveness of nominal, Delphi and interacting group decision making processes. Acad. Manage. J. 1974, 17(4):605-621.
111. Gustafson D.H., Shukla R.K., Delbecq D., Walster G.W. A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organ. Behav. Hum. Perform. 1973, 9:280-291.
112. Parent R.J., Anderson J.K., Myers P., O'Brien T. An examination of the factors contributing to Delphi accuracy. Int. J. Forecasting 1984, 3:173-182.
113. Erffmeyer R.C., Lane I.M. Quality and acceptance of an evaluative task: the effects of four group decision-making formats. Group Organ. Stud. 1984, 9(4):509-529.
114. Sniezek J. A comparison of techniques for judgmental forecasting by groups with common information. Group Organ. Stud. 1990, 15(1):5-19.
115. Van Dijk J.M.G.M. Delphi questionnaires versus individual and group interviews. Technol. Forecast. Soc. Change 1990, 37:293-304.
116. Rowe G., Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecasting 1999, 15:353-375.
117. Simoens S. Using the Delphi technique in economic evaluation: time to revisit the oracle?. J. Clin. Pharm. Ther. 2006, 31:519-522.
118. Bowles N. The Delphi technique. Nurs. Stand. 1999, 13(45):32-36.
119. Day J., Bobeva M. Seeking the truth: the use of Delphi studies for IS research. Reflection on The Past, Making Sense of Today and Predicting the Future of Information Systems, 9th Annual UKAIS Conference Proceedings Annual Conference 2004, Glasgow Caledonian University, Glasgow. K. Grant, D.A. Edgar, M. Jordan (Eds.).
120. Krefting L. Rigor in qualitative research: the assessment of trustworthiness. Am. J. Occup. Ther. 1991, 45(3):214-222.
121. Engles T.C.E., Kennedy H.P. Enhancing a Delphi study on family-focused prevention. Technol. Forecast. Soc. Change 2007, 74(4):433-451.
122. Kennedy P.H. Enhancing Delphi research: methods and results. J. Adv. Nurs. 2004, 45(5):504-511.
123. Skulmoski G.J., Hartman F.T., Krahn J. The Delphi method for graduate research. J. Inf. Technol. Educ. 2007, 6. http://informingscience.org/jite/documents/Vol6/JITEv6p001-021Skulmoski212.pdf.
124. Efstathiou N., Ameen J., Coll A.-M. A Delphi study to identify healthcare users' priorities for cancer care in Greece. Eur. J. Oncol. Nurs. 2008, 12(4):362-371.
125. Mitchell W.V. The Delphi technique: an exposition and application. Technol. Anal. Strateg. Manag. 1991, 3:333-358.
126. Nambisan S., Agarwal R., Tanniru M. Organisational mechanisms for enhancing user innovation in information technology. MIS Quart. 1999, 23(8):365-395.
127. Wynekoop J.L., Walz D.B. Investigating traits of top performing software developers. Inf. Technol. People 2000, 13(3):186-197.
128. Keil M., Tiwana A., Bush A. Reconciling user and project manager perceptions of IT project risk: a Delphi study. Inform. Syst. J. 2002, 12(2):103-119.
129. Niederman F., Brancheau J.C., Wetherbe J.C. Information systems management issues for the 1990s. MIS Quart. 1991, 15(4):475-500.
130. Delbecq A.L., Van de Ven A.H., Gustafson D.H. Group techniques for program planning. A Guide to Nominal Group and Delphi Processes 1975, Scott, Foresman and Company, Glenview, IL.
131. Zanger B.R.K., Groves D.L. A framework for the analysis of theories of sport and leisure management. J. Soc. Behav. Pers. 1994, 22:57-68.
132. De Loe R.C. Exploring complex policy questions using the policy Delphi. A multi-round, interactive survey method. Appl. Geogr. 1995, 15:55-68.
133. Boote J., Barber R., Cooper C. Principles and indicators of successful consumer involvement in NHS research: results of a Delphi study and subgroup analysis. Health Policy 2005, 75:280-297.
134. Ali A.K. Using the Delphi technique to search for empirical measures of local planning agency power. Qual. Rep. 2005, 10(4):718-744.
135. Schmidt R., Lyytinen K., Keil M., Cule P. Identifying software project risks: an international Delphi study. J. Manage. Inform. Syst. 2001, 17(4):5-36.
136. Brungs A., Jamieson R. Identification of legal issues for computer forensics. Inform. Syst. Manage. 2005, 22(2):57-66.
137. Ayton P., Ferrell W.R., Stewart T.R. Commentaries on ;The Delphi technique as a forecasting tool: issues and analysis by Rowe and Wright. Int. J. Forecasting 1999, 15:377-381.
138. Helmer O. Systematic Use of Expert Opinions (Document Number P-3721) 1967, The RAND Corporation, Santa Monica, California.