Systematic review and aggregation of empirical studies on elicitation techniques

IEEE Transactions on Software Engineering

Volumn 37, Issue 2, 2011, Pages 283-304

  Dieste, Oscar ^a   Juristo, Natalia ^a  

^a UNIVERSIDAD POLITÉCNICA DE MADRID   (Spain)

Author keywords

Elicitation methods; experimentation; performance measures; systematic literature review

Indexed keywords

SOFTWARE ENGINEERING;

COMPARATIVE ANALYSIS; ELICITATION METHODS; ELICITATION TECHNIQUES; EXPERIMENTATION; PERFORMANCE MEASURE; PROTOCOL ANALYSIS; SYSTEMATIC LITERATURE REVIEW; SYSTEMATIC REVIEW;

COMPUTER SOFTWARE;

EID: 79953167813     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2010.33     Document Type: Article

References (102)

1. L. Adelman, "Measurement Issues in Knowledge Engineering," IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 483-488, May/June 1989. (Pubitemid 20607221)
2. R. Agarwal and M.R. Tanniru, "Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation," J. Management Information Systems, vol. 7, pp. 123-141, 1990.
3. T. Bech-Larsen and N.A. Nielsen, "A Comparison of Five Elicitation Techniques for Elicitation of Attributes of Low Involvement Products," J. Economic Psychology, vol. 20, pp. 315-341, 1999.
4. E. Breivik and M. Supphellen, "Elicitation of Product Attributes in an Evaluation Context: A Comparison of Three Elicitation Techniques," J. Economic Psychology, vol. 24, pp. 77-98, 2003. (Pubitemid 36027725)
5. G.J. Browne and M.B. Rogich, "An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques," J. Management Information Systems, vol. 17, pp. 223-249, 2001. (Pubitemid 32716740)
6. A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G. Rugg, "A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1," Proc. Expert Systems '87 Research and Development in Expert Systems IV, The Ann. Technical Conf. British CS Specialist Group on Expert Systems, D.S. Moralee, ed., 1987.
7. B. Corbridge, G. Rugg, N.P. Major, N.R. Shadbolt, and A.M. Burton, "Laddering-Technique and Tool Use in Knowledge Acquisition," Knowledge Acquisition, vol. 6, pp. 315-341, 1994.
8. B. Crandall, "A Comparative Study of Think-Aloud and Critical Decision Knowledge Elicitation Methods," ACM SIGART Bull., vol. 108, pp. 144-146, 1989.
9. J.M. Moore and F.M.I. Shipman, "A Comparison of Questionnaire-Based and GUI-Based Requirements," Proc. 15th IEEE Int'l Conf. Automated Software Eng., pp. 35-43, 2000.
10. T.J. Larsen and J.D. Naumann, "An Experimental Comparison of Abstract and Concrete Representations in Systems Analysis," Information & Management, vol. 22, pp. 29-40, 1992.
11. J.E. Fowlkes, E. Salas, and D.P. Baker, "The Utility of Event-Based Knowledge Elicitation," Human Factors, vol. 42, pp. 24-35, 2000. (Pubitemid 30456526)
12. V. Goodrich and L. Olfman, "An Experimental Evaluation of Task and Methodology Variables for Requirements Definition Phase Success," Proc. 23rd Ann. Hawaii Int'l Conf. System Sciences, pp. 201-209, 1990. (Pubitemid 20702014)
13. E. Hudlicka, "Requirements Elicitation with Indirect Knowledge Elicitation: Comparison of Three Methods," Proc. Second IEEE Int'l Conf. Requirements Eng., pp. 4-11, 1996.
14. S.R. Jones and J.C. Miles, "The Use of a Prototype System for Evaluating Knowledge Elicitation Techniques," Expert Systems, vol. 15, pp. 83-97, 1998.
15. A.M. Burton, N.R. Shadbolt, G. Rugg, and A.P. Hedgecock, "The Efficacy of Knowledge Acquisition Techniques: A Comparison Across Domains and Levels of Expertise," Knowledge Acquisition, vol. 2, pp. 167-178, 1990.
16. J.W. Moody, R.P. Will, and J.E. Blanton, "Enhancing Knowledge Elicitation Using the Cognitive Interview," Expert Systems with Applications, vol. 10, pp. 127-133, 1996. (Pubitemid 126392546)
17. A.L. Rowe, N.J. Cooke, E.P. Hall, and T.L. Halgren, "Toward an On-Line Knowledge Assessment Methodology: Building on the Relationship between Knowing and Doing," J. Experimental Psychology: Applied, vol. 2, pp. 31-47, 1996. (Pubitemid 126450583)
18. R.W. Zmud, W.P. Anthony, and R.M. Stair, "The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis," J. Management Information Systems, vol. 9, pp. 175-191, 1993.
19. M. Eva, "Requirements Acquisition for Rapid Applications Development," Information & Management, vol. 39, pp. 101-107, Dec. 2001. (Pubitemid 32903860)
20. G.M. Marakas and J.J. Elam, "Semantic Structuring in Analyst and Representation of Facts in Requirements Analysis," Information Systems Research, vol. 9, pp. 37-63, 1998.
21. G. Rugg, C. Corbridge, N.P. Major, A.M. Burton, and N. Shadbolt, "A Comparison of Sorting Techniques in Knowledge Acquisition," J. Knowledge Acquisition, vol. 4, pp. 279-291, 1992.
22. M.G. Pitts and G.J. Browne, "Stopping Behavior of Systems Analysts during Information Requirements Elicitation," J. Management Information Systems, vol. 21, pp. 203-226, 2004.
23. A. Griffin and J.R. Hauser, "The Voice of the Customer," Marketing Science, vol. 12, pp. 1-27, 1993.
24. R. Schweickert, A.M. Burton, N.K. Taylor, E.N. Corlett, N.R. Shadbolt, and A.P. Hedgecock, "Comparing Knowledge Elicitation Techniques: A Case Study," Artificial Intelligence Rev., vol. 1, pp. 245-253, 1987.
25. J.A. Silver and J.C. Thompson, "Understanding Customer Needs-A Systematic Approach to the 'Voice of the Customer'," MS thesis, Massachusetts Inst. of Technology, Sloan School of Management, 1991.
26. L.A. Freeman, "The Effects of Concept Maps on Requirements Elicitation and System Models During Information Systems Development," Concept Maps: Theory, Methodology, Technology, Proc. First Int'l Conf. Concept Mapping, 2004.
27. L. Johnson and N.E. Johnson, "Knowledge Elicitation Involving Teachback Interviewing," Knowledge Acquisition for Expert Systems: A Practical Handbook, A.L. Kidd, ed., pp. 91-108, Plenum Press, 1987.
28. K.A. Ericson and H.A. Simon, Protocol Analysis: Verbal Reports as Data. Bradford Books/MIT Press, 1984.
29. M. LaFrance, "The Knowledge Acquisition Grid: A Method for Training Knowledge Engineers," Proc. Knowledge Acquisition for Knowledge-Based Systems Workshop, 1986.
30. J. Wood and D. Silver, Joint Application Development. Wiley, 1995.
31. L.J. Korhonen, "Nominal Group Technique," Adult Learning Methods, N.W. Galbraith, ed., pp. 247-259, Krieger Publishing Company, 1990.
32. A.P. Massey and W.A. Wallace, "Focus Groups as a Knowledge Elicitation Technique: An Exploratory Study," IEEE Trans. Knowledge and Data Eng., vol. 3, no. 2, pp. 193-200, June 1991. (Pubitemid 21690598)
33. L. Adelman, P.J. Sticha, and M.L. Donnel, "An Experimental Investigation of the Relative Effectiveness of Two Techniques for Structuring Multiattributed Hierarchies," Organizational Behavior and Human Decision Processes, vol. 37, pp. 188-196, 1986.
34. S. Frankel, "NGT+MDS: An Adaptation of the Nominal Group Technique for Ill-Structured Problems," J. Applied Behavioral Science, vol. 23, pp. 543-551, 1987.
35. A. Sutcliffe, "A Technique Combination Approach to Requirements Engineering," Proc. Third IEEE Int'l Symp. Requirements Eng., pp. 65-74, 1997.
36. R.R. Hoffman, "The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology," AI Magazine, vol. 8, pp. 53-67, 1987.
37. N.R. Shadbolt and A.M. Burton, "The Empirical Study of Knowledge Elicitation Techniques," ACM SIGART Bull., vol. 108, pp. 15-18, 1989.
38. R.R. Hoffman, N.R. Shadbolt, A.M. Burton, and G. Klein, "Eliciting Knowledge from Experts: A Methodological Analysis," Organizational Behavior and Human Decision Processes, vol. 62, pp. 129-158, 1995.
39. A. Blandford and G. Rugg, "A Case Study on Integrating Contextual Information with Analytical Usability Evaluation," Int'l J. Human-Computer Studies, vol. 57, pp. 75-99, 2002.
40. M.A. Musen, L.M. Fagan, D.M. Combs, and E.H. Shortliffe, "Use of a Domain Model to Drive an Interactive Knowledge-Editing Tool," Int'l J. Man-Machine Studies, vol. 26, pp. 105-121, 1987.
41. J. Coughlan and R.D. Macredie, "Effective Communication in Requirements Elicitation: A Comparison of Methodologies," Requirements Eng. J., vol. 7, pp. 47-60, 2002.
42. R. Martinez-Bejar, R. Benjamins, F. Martin, and V. Castillo, "Deriving Formal Parameters for Comparing Knowledge Elicitation Techniques Based on Mathematical Functions," Proc. 10th Knowledge Acquisition for Knowledge-Based Systems Workshop, 1996.
43. J.G. Gammack and A. Anderson, "Constructive Interaction in Knowledge Engineering," Expert Systems, vol. 7, pp. 19-26, 1990.
44. B. Fischhoff, "Eliciting Knowledge for Analytical Representation," IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 448-461, May/June 1989. (Pubitemid 20607218)
45. S.E. Gordon, K.A. Schmierer, and R.T. Gill, "Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design," Human Factors, vol. 35, pp. 459-481, 1993. (Pubitemid 23322369)
46. C.W. Holsapple and V.S. Raj, "An Exploratory Study of Two KA Methods," Expert Systems, vol. 11, pp. 77-87, 1994.
47. L.E. Wood, T.C. Davis, S.L. Clay, J.M. Ford, and S. Lammersen, "Evaluation of Interviewing Methods and Mediating Representations for Knowledge Acquisition," Int'l J. Expert Systems, vol. 8, pp. 1-23, 1995.
48. T.A. Byrd, K.L. Cossick, and R.W. Zmud, "A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques," MIS Quarterly, vol. 16, pp. 117-138, 1992.
49. M.C. Munro and G.B. Davis, "Determining Management Information News: A Comparison of Methods," MIS Quarterly, vol. 1, pp. 55-67, 1977.
50. A. Sutcliffe, "Requirements Rationales: Integrating Approaches to Requirements Analysis," Proc. Designing Interactive Systems, pp. 33-42, 1995.
51. N.J. Cooke, "Varieties of Knowledge Elicitation Techniques," Int'l J. Human-Computer Studies, vol. 41, pp. 801-849, 1994.
52. A. Hart, Knowledge Acquisition for Expert Systems. Anchor Brendon Ltd., 1986.
53. J.G. Gammack and R.M. Young, "Psychological Techniques for Eliciting Expert Knowledge," Knowledge Acquisition for Expert Systems, A. Kidd, ed., pp. 137-163, Plenum Press, 1985.
54. J.H. Boose, "A Survey of Knowledge Acquisition Techniques and Tools," Knowledge Acquisition, vol. 1, pp. 3-37, 1989.
55. N.A.M. Maiden and G. Rugg, "ACRE: Selecting Methods for Requirements Acquisition," Software Eng. J., vol. 11, pp. 183-192, 1996. (Pubitemid 126814821)
56. G. Kotonya and I. Sommerville, Requirements Engineering: Processes and Techniques. Wiley, 1998.
57. L. Macaulay, Requirements Engineering, Springer, 1996.
58. S. Lauesen, Software Requirements: Styles and Techniques. Addison-Wesley, 2002.
59. J.A. Goguen and C. Linde, "Techniques for Requirements Elicitation," Proc. IEEE Int'l Symp. Requirements Eng., pp. 152-164, 1993.
60. S. Robertson, "Requirements Trawling: Techniques for Discovering Requirements," Int'l J. Human-Computer Studies, vol. 55, pp. 405-421, 2001.
61. A.M. Hickey and A.M. Davis, "A Tale of Two Ontologies: The Basis for Systems Analysis Technique Selection," Proc. Ninth Ann. Am. Conf. Information Systems, 2003.
62. N.J. Cooke, "Eliciting Semantic Relations for Empirically Derived Networks," Int'l J. Man-Machine Studies, vol. 37, pp. 721-750, 1992.
63. B.A. Kitchenham, "Procedures for Performing Systematic Reviews," Keele Univ. Technical Report TR/SE-0401, 2004.
64. C.C. Ragin, The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Univ. of California Press, 1987.
65. R.K. Yin and K.A. Heald, "Using the Case Survey Method to Analyze Policy Studies," Administrative Science Quarterly, vol. 20, pp. 371-381, 1975.
66. A. Davis, O. Dieste, A. Hickey, N. Juristo, and A.M. Moreno, "Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review," Proc. 14th IEEE Int'l Conf. Requirements Eng., 2006.
67. P.G. Benson, S.P. Curley, and G.F. Smith, "Belief Assessment: An Underdevelopment Phase of Probability Elicitation," Management Science, vol. 41, pp. 1639-1653, 1995.
68. G.J. Browne, S.P. Curley, and P.G. Benson, "Evoking Information in Probability Assessment: Knowledge Maps and Reasoning-Based Directed Questions," Management Science, vol. 43, pp. 1-14, 1997. (Pubitemid 127575301)
69. G.P. Hodgkinson, A.J. Maule, and N.J. Bown, "Causal Cognitive Mapping in the Organizational Strategy Field: A Comparison of Alternative Elicitation Procedures," Organizational Research Methods, vol. 7, pp. 3-26, Jan. 2004.
70. R. Kanwar, J.C. Olson, and L.S. Sims, "Toward Conceptualizing and Measuring Cognitive Structures," Advances in Consumer Research, vol. 8, pp. 122-127, 1981.
71. E.W. Duggan and C.S. Thachenkary, "Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique," Information Technology and Management, vol. 4, pp. 391-408, 2003.
72. W.J. Lloyd, M.B. Rosson, and J.D. Arthur, "Effectiveness of Elicitation Techniques in Distributed Requirements Engineering," Proc. IEEE Joint Int'l Conf. Requirements Eng., pp. 311-318, 2002.
73. R. Ocker and S.R. Hiltz, "The Effects of Distributed Group Support and Process Structuring on Software Requirements Development Teams: Results on Creativity and Quality," J. Management Information Systems, vol. 12, pp. 127-154, 1996.
74. A. Sutcliffe, "Requirements Rationales: Integrating Approaches to Requirements Analysis," Proc. Designing Interactive Systems, pp. 33-42, 1995.
75. M. Tawbi, C. Ben Achour, and F. Velez, "Guiding the Process of Requirements Elicitation through Scenario," Proc. 10th Int'l Workshop Database and Expert Systems Applications, pp. 345-349, 1999.
76. G.W. Noblit and R.D. Hare, Meta-Ethnography: Synthesizing Qualitative Studies. Sage, 1988.
77. B.L. Paterson, S.E. Thorne, C. Canam, and C. Jillings, Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis. Sage, 2001.
78. M. Dixon-Woods, S. Agarwal, D. Jones, B. Young, and A. Sutton, "Synthesising Qualitative and Quantitative Evidence: A Review of Possible Methods," J. Health Services Research and Policy, vol. 10, pp. 45-53, 2005.
79. M. Mahmood, "System Development Methods-A Comparative Investigation," MIS Quarterly, vol. 11, pp. 293-311, 1987.
80. P.D. Chatzoglou, "Factors Affecting Completion of the Requirements Capture Stage of Projects with Different Characteristics," Information and Software Technology, vol. 39, pp. 627-640, 1997. (Pubitemid 127403211)
81. M. Tan, "Establishing Mutual Understanding in Systems Design: An Empirical Study," J. Management Information Systems, vol. 10, pp. 159-182, 1994.
82. J.W. Moody, J.E. Blanton, and P.H. Cheney, "A Theoretically Grounded Approach to Assist Memory Recall during Information Requirements Determination," J. Management Information Systems, vol. 15, pp. 79-89, 1998.
83. R.W. Zmud, W.P. Anthony, and R.M. Stair, "The Use of Mental Imagery as a Requirements Analysis Technique," Challenges and Strategies for Research in Systems Development, W.W. Cotterman and J.A. Senn, eds., pp. 337-350, John Wiley & Sons, 1992.
84. R. Lines, E. Breivik, and M. Supphellen, "Elicitation of Attributes: A Comparison of Preference Model Structures Derived from Two Elicitation Techniques," Proc. 24th European Marketing Academy Conf., pp. 641-656, 1995.
85. A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G.A. Rugg, "A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1," Proc. First European Workshop Knowledge Acquisition for Knowledge-Based Systems, 1987.
86. R. Lines, E. Breivik, and M. Supphellen, "Method Effects in Attribute Elicitation: A Multidimensional Comparison of Four Techniques," 1995.
87. A.M. Burton and N. Shadbolt, "A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1," Proc. SERC Workshop Knowledge Acquisition for Eng. Applications, pp. 20-28, 1987.
88. O. Dieste, A. Griman, and N. Juristo, "Developing Search Strategies for Detecting Relevant Experiments," Empirical Software Eng., vol. 14, pp. 513-539, 2009.
89. S.L. Pfleeger, "Soup or Art? The Role of Evidential Force in Empirical Software Engineering," IEEE Software, vol. 22, no. 1, pp. 66-73, Jan./Feb. 2005. (Pubitemid 40714957)
90. S.E. Straus, W.S. Richardson, P. Glasziou, and R.B. Haynes, Evidence-Based Medicine. Churchill Livingstone, 2005.
91. L.V. Hedges and I. Olkin, Statistical Methods for Meta-Analysis. Academic Press, 1985.
92. N. Juristo, A.M. Moreno, S. Vegas, and M. Solari, "In Search of What We Empirically Know about Unit Testing," IEEE Software, vol. 23, no. 6, pp. 72-80, Nov.-Dec. 2006. (Pubitemid 44824160)
93. T. Dyba, V. Kampenes, and D. Sjoberg, "A Systematic Review of Statistical Power in Software Engineering Experiments," Information and Software Technology, vol. 48, pp. 745-755, 2006. (Pubitemid 44218930)
94. N. Juristo, A.M. Moreno, and S. Vegas, "Reviewing 25 Years of Testing Technique Experiments," J. Empirical Software Eng., vol. 9, pp. 7-44, 2004.
95. P.G. Shekelle, M.L. Hardy, S.C. Morton, M. Maglione, W.A. Mojica, M.J. Suttorp, S.L. Rhodes, L. Jungvig, and J. Gagne, "Efficacy and Safety of Ephedra and Ephedrine for Weight Loss and Athletic Performance: A Meta-Analysis," J. Am. Medical Assoc., vol. 289, pp. 1537-1545, Mar. 2003. (Pubitemid 37430293)
96. D. Moher, A.R. Jadad, G. Nichol, M. Penman, P. Tugwell, and S. Walsh, "Assessing the Quality of Randomized Controlled Trials: An Annotated Bibliography of Scales and Checklists," Controlled Clinical Trials, vol. 16, pp. 62-73, Feb. 1995.
97. J.P.T. Higgins and S. Green, Cochrane Handbook for Systematic Reviews of Interventions. John Wiley and Sons, 2006.
98. O.P. Brereton, B. Kitchenham, D. Budgen, M. Turner, and M. Khalil, "Employing Systematic Literature Review: An Experience Report," Technical Report TR/05-01, School of Computing & Math., Keele Univ., 2005.
99. B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg, "Preliminary Guidelines for Empirical Research in Software Engineering," IEEE Trans. Software Eng., vol. 28, no. 8, pp. 721-734, Aug. 2002.
100. W. Shadish, T. Cook, and D.T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, 2001.
101. Construx Software, "Context Free Requirements Questions," http://www.construx.com/Page.aspx?hid=1742, 2008.
102. D.T. Campbell, "Reforms as Experiments," Am. Psychologist, vol. 24, pp. 409-429, 1969.