The effectiveness of software development technical reviews: A behaviorally motivated program of research

IEEE Transactions on Software Engineering

Volumn 26, Issue 1, 2000, Pages 1-14

  Sauer, Chris ^a   Ross Jeffery, D ^b   Land, Lesley ^b   Yetton, Philip ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Behavioral research; Defect detection; Defects; Expertise; Group process; Group size; Groups; Inspections; Reading; Research program; Technical reviews; Theory; Training; Walkthroughs

Indexed keywords

SOFTWARE DEVELOPMENT TECHNICAL REVIEW (SDTR);

COMPUTER SOFTWARE SELECTION AND EVALUATION; PROGRAM DEBUGGING;

SOFTWARE ENGINEERING;

EID: 0033873375     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/32.825763     Document Type: Article

References (88)

1. A.F. Ackerman, P.J. Fowler, and R. Ebenau, "Software Inspections and the Industrial Production of Software," Software Validation, North-Holland, 1984.
2. ANSI/IEEE, An American National Standard: IEEE Standard for Software Reviews and Audits, ANSI/IEEE Std 1028-1988, New York: IEEE, 1989.
3. V.R. Basili, G. Caldiera, F. Lanubile, and F. Shull, "Studies on Reading Techniques," Proc. 21st Ann. Software Eng. Workshop, Dec. 1996.
4. V.R. Basili, S. Green, O. Laitenburger, F. Lanubile, F. Shull, S. Sorumgard, and M. Zelkowitz, "The Empirical Investigation of Perspective-Based Reading," J. Empirical Software Eng., vol. 1, no. 2, 1996.
5. D.B. Bisant and J.R. Lyle, "A Two-Person Inspection Method to Improve Programming Productivity," IEEE Trans. Software Eng., vol. 15, no. 10, pp. 1,294-1,304, Oct. 1989.
6. P.C. Bottger and P.W. Yetton, "Improving Group Performance by Training in Individual Problem Solving," J. Applied Psychology, vol. 72, no. 4, pp. 651-657, 1987.
7. P.C. Bottger and P.W. Yetton, "An Integration of Process and Decision Scheme Explanations of Group Problem Solving Performance," Organizational Behavior and Human Decision Processes, vol. 42, pp. 234-249, 1988.
8. M.G. Bradac, D.E. Perry, and L.G. Votta, "Prototyping a Process Monitoring Experiment," Proc. 15th Int'l Conf. Software Eng., May 1993.
9. T.A. Corbi, "Program Understanding: Challenge for the 1990s," IBM Systems J., vol. 28, no. 2, pp. 294-306, 1989.
10. A. Diehl and W. Stroebe, "Productivity Loss in Brainstorming Groups: Toward the Solution of a Riddle," J. Personality and Social Psychology, vol. 53, no. 3, pp. 497-509, 1987.
11. S.G. Eick, C.R. Loader, M.D. Long, L.G. Votta, and S. Vander Wiel, "Estimating Fault Content before Coding," Proc. 14th Int'l Conf. Software Eng., May 1992.
12. H.J. Einhorn, R.M. Hogarth, and E. Klempner, "Quality of Group Judgement," Psychological Bulletin, vol. 84, pp. 158-172, 1977.
13. M.E. Fagan, "Design and Code Inspections to Reduce Errors in Program Development," IBM Systems }., vol. 15, no. 3, pp. 182-211, 1976.
14. M.E. Fagan, "Advances in Software Inspections," IEEE Trans. Software Eng., vol. 12, no. 7, pp. 744-751, July 1986.
15. P.). Fowler, "In-Process Inspections of Workproducts at AT&T," AT&T Technical }., pp. 102-112, Mar./Apr. 1986.
16. L.A. Franz and J.C. Shih, "Estimating the Value of Inspections for Early Testing of Software Projects," Hewlett-Packard J., vol. 45, no. 6, Dec. 1994.
17. D.P. Freedman and G.M. Weinberg, Handbook of Walkthroughs, Inspections, and Technical Reviews: Evaluating Programs, Projects, and Products, third ed. Little Brown & Co., 1982.
18. R.D. Galliers, Y. Merali, and L. Spearing, "Managing Information Technology? How British Executives Perceive the Key Issues," J. Information Technology, vol. 9, no. 1, pp. 223-238, 1994.
19. D.C. Ganster, P. Poppler, and S. Williams, "Does Training in Problem Solving Improve the Quality of Group Decisions?" J. Applied Psychology, vol. 76, no. 3, pp. 479-483, 1991.
20. T. Gilb, Principles of Software Engineering Management. Wokingham: Addison-Wesley, 1988.
21. R. Glazer, J.H. Steckel, and R.S. Winer, "Group Process and Decision Performance in a Simulated Marketing Environment," J. Business Research, vol. 15, pp. 545-557, 1987.
22. R.B. Grady, Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J.: Prentice Hall, 1992.
23. R.B. Grady and T. Van Slack, "Key Lessons in Achieving Widespread Inspection Use," IEEE Software, vol. 11, no. 4, July 1994.
24. H.A. Gurnee, "A Comparison of Collective and Group Judgements of Fact," J. Experimental Psychology, vol. 3, pp. 437-444,1937.
25. J.R. Hackman and CG. Morris, "Group Tasks, Group Interaction Process and Group Performance Effectiveness: A Review and Partial Integration," Advances in Experimental Social Psychology, L. Berkowitz, ed., vol. 8, pp. 47-99, New York: Academic Press, 1975.
26. J. Hall and W.H. Watson, "The Effects of a Normative Intervention on Group Decision-Making Performance," Human Relations, vol. 23, pp. 299-317, 1970.
27. R.A. Henry, "Improving Group Judgement Accuracy: Information Sharing and Determining the Best Member," Organizational Behavior and Human Decision Processes, vol. 62, no. 2, pp. 190-197, May 1995.
28. C.R. Holloman and H.W. Hendrick, "Individual versus Group Effectiveness in Solving Factual and Nonfactual Problems," Proc. 78th Ann. Convention, Am. Psychological Assoc., 1970.
29. P.M. Johnson and D. Tjahjono, "Assessing Software Review Meetings: A Controlled Experimental Study Using CSRS," Technical Report 96-06, Dept. of Information and Computer Sciences, Univ. of Hawaii, 1996.
30. L.P.W. Kim, C. Sauer, and R. Jeffery, "A Critical Survey of Software Development Technical Reviews as a Non-MethodSpecific Approach to Software Quality Assurance," ITRC Report #94/34, School of Information Systems, Univ. of New South Wales, Sydney, 1994.
31. J.C. Knight and A.N. Myers, "An Improved Inspection Technique," Comm. ACM, vol. 36, no. 11, pp. 51-61, Nov. 1993.
32. I. Lakatos, "Falsification and the Methodology of Scientific Research Programmes," Criticism and the Growth of Knowledge, I. Lakatos and A. Musgrave, eds., Cambridge U.K.: Cambridge Univ. Press, 1974.
33. F. Lanubile and G. Visaggio, "Assessing Defect Detection Methods for Software Requirements Inspections through External Replication," Technical Report ISERN-96-01, Int'l Software Eng. Research Network, Jan. 1996.
34. L.P.W. Land, C. Sauer, and R. Jeffery, "Validating the Defect Detection Performance Advantage of Group Designs for Software Reviews: Report of a Laboratory Experiment Using Program Code," Proc. Sixth European Software Eng. Conf.-ESEC/FSE '97, pp. 294-309, 1997.
35. P.R. Laughlin, N.L. Kerr, J.H. Davis, H.M. Halff, and K.A. Marcinak, "Group Size, Member Ability, and Social Decision Schemes on an Intellective Task," J. Personality and Social Psychology, vol. 31, no. 3, pp. 522-535, 1975.
36. R.E. Levine and R.L. Moreland, "Progress in Small Group Research," Ann. Review of Psychology, vol. 1, 1990.
37. R. Libby and R.K. Blashfield, "Performance of a Composite as a Function of the Number of Judges," Organizational Behavior and Human Performance, vol. 21, pp. 121-129, 1978.
38. R. Libby, K.T. Trotman, and I. Zimmer, "Member Variation, Recognition of Expertise, and Group Performance" J. Applied Psychology, vol. 72, pp. 81-87, 1987.
39. Generalising from Laboratory to Field Settings: Research Findings from Industrial-Organizational Psychology, Organizational Behaviour, and Human Resource Management, E.A. Locke, ed., Lexington, Mass.: Heath-Lexington, 1985.
40. I. Lorge and H. Solomon, "Two Models of Group Behaviour in the Solution of Eureka Type Problems," Psychometrica, vol. 20, pp. 139-148, 1955.
41. I. Lorge and H. Solomon, "Individual Performance and Group Performance in Problem Solving Related to Group Size and Previous Exposure to the Problem," J. Psychology, vol. 48, pp. 107-114, 1959.
42. I. Lorge and H. Solomon, "Group and Individual Performance in Problem Solving Related to Previous Exposure to the Problem, Level of Aspiration, and Group Size," Behavioral Science, vol. 5, pp. 28-38, 1960.
43. I. Lorge, D. Fox, J. Davitz, and M. Brenner, "A Survey of Studies Contrasting the Quality of Group Performance and Individual Performance," Psychological Bulletin, vol. 55, pp. 337-371, 1958.
44. F.J. Lukey, "Understanding and Debugging Programs," Int'l J. Man-Machine Studies, vol. 12, pp. 189-202, 1980.
45. F. Macdonald, J. Miller, A. Brooks, M. Roper, and M. Wood, "Automating the Software Inspection Process," Proc. Seventh Int'l Workshop Computer-Aided Software Eng., CASE '95, July 1995.
46. R. Madachy, L. Little, and S. Fan, "Analysis of a Successful Inspection Program," Proc. Software Eng. Workshop, SEW, SEL-93-003, pp. 176-188, 1993.
47. J. Martin and W.T. Tsai, "N-fold Inspection: A Requirements Analysis Technique," Comm. ACM, vol. 33, no. 2, pp. 225-232, Feb. 1990.
48. V. Mashayekhi, J.M. Drake, W.T. Tsai, and J. Riedl, "Distributed, Collaborative Software Inspections," IEEE Software, pp. 66-75, Sept. 1993.
49. L.K. Michaelsen, W.E. Watson, and R.H. Black, "Realistic Test of Individual versus Group Decision Making," J. Applied Psychology, vol. 74, pp. 834-839, 1989.
50. G.J. Myers, "A Controlled Experiment in Program Testing and Code Walkthroughs/Inspections," Comm. ACM, vol. 21, no. 9, pp. 760-768, Sept. 1978.
51. F. Niederman, J.C. Brancheau, and J.C. Wetherbe, "Information Systems Management Issues for the 1990s," MIS Quarterly, pp. 475-500, Dec. 1991.
52. G.M. Olson, S. Sheppard, and E. Soloway, Empirical Studies of Programmers: Second Workshop. Norwood, N.J.: Ablex Publishing, 1987.
53. D.L. Parnas and D.M. Weiss, "Active Design Reviews: Principles and Practices," Proc. Eighth Int'l Conf. Software Eng., pp. 215-222, Aug. 1985.
54. N. Pennington, "Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs," Cognitive Psychology, vol. 19, pp. 295-341, 1987.
55. A.A. Porter, H. Siy, C.A. Toman, and L.G. Votta, "An Experiment to Assess the Cost-Benefits of Code Inspections in Large-Scale Software Development," http://www.cs.umd.edu/~aporter/ live.ps., 1996.
56. A.A. Porter and L.G. Votta, "An Experiment to Assess Different Defect Detection Methods for Software Requirements Inspections," Proc. 16th Int'l Conf. Software Eng., pp. 103-112, May 1994.
57. A.A. Porter, L.G. Votta, and V.R. Basili, "Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment," IEEE Trans. Software Eng., vol. 21, no. 6, pp. 563-575, June 1995.
58. Z.W. Pylyshyn, Computation and Cognition: Toward a Foundation for Cognitive Science. Cambridge, Mass: Bradford Books, MIT Press, 1984.
59. F. Redmill, "Pagan's Inspection: Achieving Quality in Code and Documentation; Built-in Gauge of Effectiveness," Managing System Development, vol. 13, no. 3, pp. 1-5, Mar. 1993.
60. G.M. Schneider, J. Martin, and W.T. Tsai, "An Experimental Study of Fault Detection in User Requirements Documents," ACM Trans. Software Eng. and Methodology, vol. 1, no. 2, pp. 188-204, Apr. 1992.
61. M.E. Shaw, "Comparison of Individuals and Small Groups in the Rational Solution of Complex Problems," Am. J. Psychology, vol. 44, pp. 491-504, 1932.
62. M.E. Shaw, Group Dynamics: The Psychology of Small Group Behavior. New York: McGraw-Hill, 1981.
63. H.P. Siy, "Identifying the Mechanisms Driving Code Inspection Costs and Benefits," PhD thesis, Univ. of Maryland, 1996 (unpublished).
64. Empirical Studies of Programmers, E. Soloway and S. lyengar, eds. Norwood, N.J.: Ablex Publishing, 1986.
65. I.D. Steiner, Group Process and Productivity. New York: Academic Press, 1972.
66. R.G. Strauss and R.G. Ebenau, Software Inspection Process. New York: McGraw-Hill, 1994.
67. R.L. Thorndike, "In What Type of Task Does the Group Do Well?" J. Abnormal and Social Psychology, vol. 33, pp. 408-412, 1938.
68. K.T. Trotman, "The Review Process and the Accuracy of Auditor Judgements," J. Accounting Research, vol. 23, no. 2, pp. 740-752, 1985.
69. K.T. Trotman and P.W. Yetton, "The Effect of the Review Process on Auditor Judgements," J. Accounting Research, vol. 23, no. 1, pp. 256-267, 1985.
70. K.T. Trotman, P.W. Yetton, and I.R. Zimmer, "Group Size and Performance: Prediction of Failure by Loan Officers," Australian J. Management, pp. 127-136, 1981.
71. K.T. Trotman, P.W. Yetton, and I.R. Zimmer, "Individual and Group Judgements of Internal Control Systems," J. Accounting Research, vol. 21, no. 1, pp. 286-292, 1983.
72. L.G. Votta, "Does Every Inspection Need a Meeting?" Proc. First ACM SIGSOFT Symp. Foundations of Software Eng., Software Eng. Notes, D. Notkin, ed., vol. 18, no. 5, pp. 107-114, Dec. 1993.
73. W. Watson, W. Sharp, and L.K. Michaelsen, "Member Competence, Group Interaction, and Group Decision Making: A Longitudinal Study," J. Applied Psychology, vol. 76, no. 6, pp. 803-809, 1991.
74. G.M. Weinberg and D.P. Freedman, "Reviews, Walkthroughs, and Inspections," IEEE Trans. Software Eng., vol. 10, no. 1, pp. 68-72, Jan. 1984.
75. E.F. Weller, "Lessons from Three Years of Inspection Data," IEEE Software, pp. 38-45, Sept. 1993.
76. P.W. Yetton and P.C. Bottger, "Individual versus Group Problem Solving: An Empirical Test of a Best-Member Strategy," Organizational Behavior and Human Performance, vol. 29, no. 3, pp. 307-321, June 1982.
77. P.W. Yetton and P.C. Bottger, "The Relationship among Group Size, Member Ability, Social Decision Schemes, and Performance," Organizational Behavior and Human Performance, vol. 32, pp. 145-159, 1983.
78. E. Yourdon, Structured Walkthroughs, fourth ed. Prentice Hall, 1989.
79. R.C. Ziller, "Group Size: A Determinant of the Quality and Stability of Group Decisions," Sociometry, vol. 20, pp. 165-173, 1957.
80. S. Rifkin and L. Deimel, "Applying Program Comprehension Techniques to Improve Software Inspections," Proc. 19th Ann. NASA Software Laboratory Workshop, Nov. 1994.
81. L. Land, C. Sauer, and R. Jeffery, "The Performance Effects of Process Roles in Code Reviews: A Preliminary Empirical Investigation," Caesar Technical Report 97/7, Univ. of New South Wales, School of Information Systems, Sydney, 1997.
82. L.P.W. Land, R. Jeffery, and C. Sauer, "Validating the Defect Detection Performance Advantage of Group Designs for Software Reviews: Report of a Replicated Experiment," Proc. Australian Software Eng. Conf., P.A. Baues ed., 1997.
83. S.H. Kan, V.R. Basili, and L.N. Shapiro, "Software Quality: An Overview from the Perspective of Total Quality Management," IBM Systems J., vol. 33, no. 1, pp. 4-18, 1994.
84. E. Yourdon, "Quality: What It Means and How to Achieve It," Management Information Science, pp. 43-47, Feb. 1993.
85. M.C. Paulk, B. Vurtis, and M.B. Chrissis, "Capability Maturity Model, version 1.1," IEEE Software, pp. 18-27, July 1993.
86. A.K. Onoma and T. Yamaura, "Practical Steps Toward Quality Development," IEEE Software, pp. 68-76, Sept. 1995.
87. P. Hsia, "Learning to Put Lessons into Practice," IEEE Software, pp. 14-17, Sept. 1993.
88. T. Gilb and D. Graham, Software Inspection. Harlow, Essex: Addison-Wesley, 1993.