An empirical review of software process assessments

Advances in Computers

Volumn 53, Issue C, 2000, Pages 319-423

  Emam, Khaled El ^a   Goldenson, Dennis R ^b  

^a NATIONAL RESEARCH COUNCIL CANADA   (Canada)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 23044518225     PISSN: 00652458     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0065-2458(00)80008-X     Document Type: Review

References (170)

1. Krasner H. The payoff for software process improvement: what it is and how to get it. In: El Emam K., and Madhavji N.H. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press, New York
2. available from. McGibbon T. A business case for software process improvement revised: measuring return on investment from software engineering and management. DoD Data and Analysis Center for Software (DACS), Technical Report September (1999). www.dacs.dtic.mil/techs/roispi2/roispi2.pdf
3. Card D. Understanding process improvement. IEEE Software July (1991) 102-103
4. Technical Report SEL-95-002. Software Engineering Laboratory. Software Process Improvement Guidebook (1995), NASA/GSFC
5. Software Engineering Institute. The Capability Maturity Model: Guidelines for Improving the Software Process (1995), Addison-Wesley, Reading, MA
6. El Emam K., Drouin J.-N., and Melo W. SPICE: Theory and Practice of Software Process Improvement and Capability Determination (1998), IEEE CS Press, New York
7. Nolan R. Managing the computer resource: a stage hypothesis. Communications of the ACM 16 7 (1973) 399-405
8. Dunaway D., and Masters S. CMM-based appraisal for internal process improvement (CBA IPI): method description. Technical Report CMU/SEI-96-TR-007 (1996), Software Engineering Institute
9. Nolan R. Managing the crisis in data processing. Harvard Business Review March/April (1979) 115-126
10. Ahituv N., and Neumann S. Principles of Information Systems for Management (1982), W. Brown
11. Alter S. Decision Support Systems: Current Practices and Continuing Challenges (1980), Addison-Wesley, Reading, MA
12. Ein-Dor P., and Segev E. Managing Management Information Systems (1978), Lexington
13. Keen P., and Scott Morton M. Decision Support Systems: An Organizational Perspective (1978), Addison-Wesley, Reading, MA
14. Gibson C., and Nolan R. Behavioral and organizational issues in the stages of managing the computer resource. In: Nolan R. (Ed). Managing the Data Resource Function (1974), West Publishing Company
15. Gibson C., and Nolan R. Managing the four stages of EDP growth. Harvard Bussiness Review (1974) 76-88
16. Nolan R. Thoughts about the fifth stage. Database 7 2 (1975) 4-10
17. Benbasat I., Dexter A., and Mantha R. Impact of organizational maturity on information system skill needs. MIS Quarterly 4 1 (1980) 21-34
18. Lucas H., and Sutton J. The stage hypothesis and the S-curve: some contradictory evidence. Communications of the ACM 20 4 (1977) 254-259
19. Goldstein R., and McCririck I. The stage hypothesis and data administration: some contradictory evidence. Proceedings of the 2nd International Conference on Information Systems (1981) 309-324
20. Drury D. An empirical assessment of the stages of data processing growth. MIS Quarterly 7 2 (1983) 59-70
21. Benbasat I., Dexter A., Drury D., and Goldstein R. A critique of the stage hypothesis: theory and empirical evidence. Communications of the ACM 27 5 (1984) 476-485
22. King J., and Kraemer K. Evolution and organizational information systems: an assessment of Nolan's stage model. Communications of the ACM 27 5 (1984) 466-475
23. Huff S., Munro M., and Martin B. Growth stages of end user computing. Communications of the ACM 31 5 (1988) 542-550
24. Gregoire J.-F., and Lustman F. The stage hypothesis revisited: an EDP professionals' point of view. Information and Management 24 (1993) 237-245
25. Humphrey W. Characterizing the software process: a maturity framework. IEEE Software March (1988) 73-79
26. Radice R., Harding J., Munnis P., and Phillips R. A programming process study. IBM Systems Journal 24 2 (1985) 91-101
27. Paulk M., Weber C., and Chrissis M.-B. The capability maturity model for software. In: El Emam K., and Madhavji N.H. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
28. Humphrey W., and Sweet W. A method for assessing the software engineering capability of contractors. Technical Report CMU/SEI-87-TR-23 (1987), Software Engineering Institute
29. Drouin J.-N. Introduction to SPICE. In: El Emam K., Drouin J.-N., and Melo W. (Eds). SPICE: Theory and Practice of Software Process Improvement and Capability Determination (1998), IEEE CS Press
30. Coallier F., Mayrand J., and Lague B. Risk management in software product procurement. In: El Emam K., and Madhavji N. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
31. Craigmyle M., and Fletcher I. Improving IT effectiveness through software process assessment. Software Quality Journal 2 (1993) 257-264
32. Sommerville I., and Sawyer P. Requirements Engineering: A Good Practice Guide (1997), John Wiley, New York
33. Burnstein I., Suwannasart T., and Carlson C. Developing a testing maturity model: Part I. Crosstalk August (1996) 21-24
34. Burnstein I., Suwannasart T., and Carlson C. Developing a testing maturity model: Part II. Crosstalk September (1996) 19-24
35. Budlong F., and Peterson J. Software Metrics Capability Evaluation Guide (1995), Software Technology Support Center, Ogden Air Logistics Center, Hill Air Force Base
36. Software Engineering Institute. A Systems Engineering Capability Maturity Model, Version 1.0. Handbook CMU/SEI-94-HB-04 (1994), Software Engineering Institute
37. Curtis B., Hefley W., Miller S., and Konrad M. The people capability maturity model for improving the software workforce. In: El Emam K., and Madhavji N. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
38. Drew D. Tailoring the Software Engineering Institute's (SEI) Capability Maturity Model (CMM) to a software sustaining engineering organization. Proceedings of the International Conference on Software Maintenance (1992) 137-144
39. Jones C. The economics of software process improvements. In: El Emam K., and Madhavji N.H. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
40. Bootstrap Project Team. Bootstrap: Europe's Assessment Method. IEEE Software May (1993) 93-95
41. Haase V., Messnarz R., Koch, et al. Bootstrap: fine-tuning process assessment. IEEE Software July (1994) 25-35
42. Steinen H. Software process assessment and improvement: 5 years of experiences with Bootstrap. In: El Emam K., and Madhavji N. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
43. Paulk M. The evolution of the SEI's capability maturity model for software. Software Process-Improvement and Practice. Pilot Issue (1995) 1-15
44. Information Technology-Software Process Assessment (1998). www.seq.itt.nrc.ca/spice
45. Herbsleb J., Zubrow D., Goldenson D., et al. Software quality and the capability maturity model. Communications of the ACM 40 6 (1997) 30-40
46. Hersh A. Where's the return on process improvement?. IEEE Software July (1993) 12
47. Fenton N. Objectives and context of measurement/experimentation. In: Rombach H.D., Basili V., and Selby R. (Eds). Experimental Software Engineering Issues: Critical Assessment and Future Directions (1993), Springer-Verlag, Berlin
48. Jones C. Gaps in SEI programs. Software Development 3 3 (1995) 41-48
49. Gray E., and Smith W. On the limitations of software process assessment and the recognition of a required re-orientation for global process improvement. Software Quality Journal 7 (1998) 21-34
50. Pfleeger S.-L. Understanding and improving technology transfer in software engineering. Journal of Systems and Software 47 (1999) 111-124
51. Benbasat I., and Zmud R. Empirical research in information systems: the practice of relevance. MIS Quarterly 23 1 (1999) 3-16
52. available at. Herbsleb J. Hard problems and hard science: on the practical limits of experimentation. IEEE TCSE Software Process Newsletter 11 (1998) 18-21. www.seq.iit.nrc.ca/spn
53. Raghavan S., and Chand D. Diffusing software-engineering methods. IEEE Software July (1989) 81-90
54. Weissfelner S. ISO 9001 for software organizations. In: El Emam K., and Madhavji N. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
55. McFeeley B. IDEAL: a user's guide for software process improvement. Technical Report CMU/SEI-96-HB-001 February (1996), Software Engineering Institute
56. Paulk M., and Konrad M. Measuring process capability versus organizational process maturity. Proceedings of the 4th International Conference on Software Quality. October (1994)
57. Dunaway D. CMM-Based Appraisal for Internal Process Improvement (CBA IPI) Lead Assessor's Guide. Handbook CMU/SEI-96-HB-003 (1996), Sofware Engineering Institute
58. Kitson D., and Masters S. An analysis of SEI software process assessment results: 1987-1991. Proceedings of the International Conference on Software Engineering (1993) 68-77
59. Simon J.-M. Assessment using SPICE: a case study. In: El Emam K., Drouin J.-N., and Melo W. (Eds). SPICE: Theory and Practice of Software Process Improvement and Capability Determination (1998), IEEE CS Press
60. Maclennan F., Ostrolenk G., and Tobin M. Introduction to the SPICE Trials. In: El Emam K., Drouin J.-N., and Melo W. (Eds). SPICE: Theory and Practice of Software Process Improvement and Capability Determination (1998), IEEE CS Press
61. Klein D. If you get straight A's, you must be intelligent-respecting the intent of the capability maturity model. Crosstalk February (1998) 22-23
62. Dymond K. Essence and accidents in SEI-style assessments or "maybe this time the voice of the engineer will be heard. In: El Emam K., and Madhavji N. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
63. Saiedian H., and Kuzara R. SEI capability maturity model's impact on contractors. IEEE Computer January (1995) 16-26
64. Rugg D. Using a capability evaluation to select a contractor. IEEE Software July (1993) 36-45
65. Dadoun G. ISO 9000: a reqirement for doing business. Proceedings of the CAS (Centre for Advanced Studies) Conference (1992), IBM Canada Ltd. 433-437
66. Huebner A. ISO 9000 implementation in software development of IBM Germany. Paper presented at SDC 1992, IBM, Application development Germany. 7 May (1992)
67. Herbsleb J., and Grinter R. Conceptual simplicity meets organizational complexity: case study of a corporate metrics program. Proceedings of the 20th International Conference on Software Engineering (1998) 271-280
68. Dunaway D., Goldenson D., Monarch I., and White D. How well is CBA IPI working? User feedback. Proceedings of the 1998 Software Engineering Process Group Conference (1998)
69. Rubin D. Multiple Imputation for Nonresponse in Surveys (1987), John Wiley, New York
70. SPIRE Project. The SPIRE Handbook: Better Faster Cheaper Software Development in Small Companies. ESSI Project 23873 November (1998)
71. El Emam K., and Goldenson D. An empirical evaluation of the prospective international SPICE standard. Software Process-Improvement and Practice 2 (1996) 123-148
72. Goldenson D.R., and Herbsleb J.D. After the appraisal: a systematic survey of process improvement, its benefits, and factors that influence success. Technical Report, CMU/SEI-95-TR-009 (1995), Software Engineering Institute
73. Drehmer D., and Dekleva S. Measuring software engineering maturity: a Rasch calibration. Proceedings of the International Conference on Information Systems (1993) 191-202
74. Buchman C. Software process improvement at AlliedSignal Aerospace. Proceedings of the 29th Annual Hawaii International Conference on Systems Science, Vol. 1: Software Technology and Architecture (1996) 673-680
75. El Emam K., Smith B., and Fusaro P. Success factors and barriers for software process improvement: an empirical study. In: Messnarz R., and Tully C. (Eds). Better Software Practice for Business Benefit: Principles and Experiences (1999), IEEE CS Press
76. Siegel S., and Castellan N.J. Nonparametric Statistics for the Behavioral Sciences (1988), McGraw-Hill, New York
77. Stelzer D., and Mellis W. Success factors of organizational change in software process improvement. Software Process-Improvement and Practice 4 4 (1998) 227-250
78. Brodman J., and Johnson D. Return on investment (ROI) from software process improvement as measured by US industry. Software Process: Improvement and Practice. Pilot Issue (1995) 35-47
79. Humphrey W., Snyder T., and Willis R. Software process improvement at Hyghes Aircraft. IEEE Software July (1991) 11-23
80. Wohlwend H., and Rosenbaum S. Software improvements in an international company. Proceedings of the International Conference on Software Engineering (1993) 212-220
81. Brodman J., and Johnson D. Return on investment from software process improvement as measured by US industry. Crosstalk 9 4 (1996) 23-29
82. Dion R. Process improvement and the corporate balance sheet. IEEE Software 10 4 (1993) 28-35
83. Hayes W., and Zubrow D. Moving on up: data and experience doing CMM-based process improvement. Technical Report CMU/SEI-95-TR-008 (1995), Software Engineering Institute
84. October. Rahhal S. An effort estimation model for implementing ISO 9001 in software organizations. Master's thesis (1995), School of Computer Science, McGill University
85. Kim J., and Mueller C. Factor Analysis: Statistical Methods and Practical Issues (1978), Sage, Thousand Oaks, CA
86. Curtis B. The factor structure of the CMM and other latent issues. Paper presented at the Empirical Studies of Programmers: 6th Workshop. Washington DC (1996)
87. April. Clark B. The effects of software process maturity on software development effort. PhD thesis (1997), University of Southern California
88. El Emam K. The internal consistency of the ISO/IEC 15504 software process capability scale. Proceedings of the 5th International Symposium on Software Metrics (1998), IEEE CS Press 72-81
89. El Emam K., and Goldenson D.R. SPICE: An empiricist's perspective. Proceedings of the 2nd IEEE International Software Engineering Standards Symposium. August (1995) 84-97
90. Goldenson D., El Emam K., Herbsleb J., and Deephouse C. Empirical studies of software process assessment methods. In: El Emam K., and Madhavji N.H. (Eds). Elements of Software Process Assessment and Improvement (1999), IEEE CS Press
91. El Emam K. Benchmarking kappa: interrater agreement in software process assessments. Empirical Software Engineering: An International Journal 4 (1999), Academic Publishers 113-133
92. To appear. El Emam K., and Garro I. Estimating the extent of standards use: the case of ISO/IEC 15504. Journal of Systems and Software (2000)
93. Bollinger T., and McGowan C. A critical look at software capability evaluations. IEEE Software, July (1991) 25-41
94. El Emam K., Smith B., and Fusaro P. Modeling the reliability of SPICE based assessments. Proceedings of the 3rd IEEE International Software Engineering Standards Symposium (1997) 69-82
95. Fusaro P., El Emam K., and Smith B. The internal consistencies of the 1987 SEI Maturity Questionnaire and the SPICE Capability Dimension. Empirical Software Engineering: An International Journal 3 (1997) 179-201
96. Whitney R., Nawrocki E., Hayes W., and Siegel J. Interim profile: development and trial of a method to rapidly measure software engineering maturity status. Technical Report, CMU/SEI-94-TR-4 (1994), Software Engineering Institute
97. Carmines E., and Zeller R. Reliability and Validity Assessment (1979), Sage, Beverly Hills, CA
98. Card D. Capability evaluations rated highly variable. IEEE Software September (1992) 105-106
99. Springsteen B., Brykczynski B., Fife D., et al. Policy assessment for the software process maturity model. Report IDA-D-1202 (1992), Institute for Defense Analysis
100. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 16 3 (1951) 297-334
101. El Emam K., Simon J.-M., Rousseau S., and Jacquet E. Cost implications of interrater agreement for software process assessments. Proceedings of the 5th International Symposium on Software Metrics (1998), IEEE CS Press 38-51
102. Lord F., and Novick M. Statistical Theories of Mental Test Scores (1968), Addison-Wesley, Reading, MA
103. Besselman J., Byrnes P., Lin C., et al. Software capability evaluations: experiences from the field. SEI Technical Review (1993)
104. Japan SC7 WG10 SPICE Committee. Report of Japanese trials process assessment by SPICE method. A SPICE Project Report (1994)
105. Masters S., and Bothwell C. CMM Appraisal Framework-Version 1.0. Technical Report CMU/SEI-TR-95-001 (1995), Software Engineering Institute
106. Nunnally J., and Bernstein I. Psychometric Theory (1994), McGraw Hill, New York
107. Subramanian A., and Nilakanta S. Measurement: a blueprint for theory building in MIS. Information and Management 26 (1994) 13-20
108. Sethi V., and King W. Construct measurement in information systems research: an illustration in strategic systems. Decision Sciences 22 (1991) 455-472
109. Ives B., Olson M., and Baroudi J. The measurement of user information satisfaction. Communications of the ACM 26 10 (1983) 785-793
110. Tait P., and Vessey I. The effect of user involvement on system success: a contingency approach. MIS Quarterly March (1988) 91-108
111. Amoako-Gyampah K., and White K. User involvement and user satisfaction: an exploratory contingency model. Information and Management 25 (1993) 1-10
112. Baroudi J., Olson M., and Ives B. An empirical study of the impact of user involvement on system usage and information satisfaction. Communications of the ACM 29 3 (1986) 232-238
113. Davis F. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly (1989) 319-340
114. Galletta D., and Ledrer A. Some cautions on the measurement of user information satisfaction. Decision Sciences 20 (1989) 419-438
115. Torkzadeh G., and Doll W. The test-retest reliability of user involvement instruments. Information and Management 26 (1994) 21-31
116. El Emam K., and Madhavji N.H. Measuring the success of requirements engineering processes. Proceedings of the 2nd IEEE International Symposium on Requirements Engineering (1995) 204-211
117. El Emam K., and Madhavji N.H. The reliability of measuring organizational maturity. Software Process: Improvement and Practice 1 1 (1995) 3-25
118. Humphrey W., and Curtis B. Comments on "a critical look.". IEEE Software July (1991) 42-46
119. Cronbach L., Gleser G., Nanda H., and Rajaratnam N. The Dependability of Behavioral Measurements: Theory of Generalizability of Scores and Profiles (1972), John Wiley, New York
120. Neter J., Wasserman W., and Kunter M. Applied Linear Statistical Models: Regression, Analysis of Variance, and Experimental Designs (1990), Irwin
121. Allen M., and Yen W. Introduction to Measurement Theory (1979), Brooks/Cole
122. El Emam K., Briand L., and Smith B. Assessor agreement in rating SPICE processes. Software Process Improvement and Practice Journal 2 4 (1996) 291-306
123. El Emam K., Goldenson D., Briand L., and Marshall P. Interrater agreement in SPICE-based assessments: some preliminary results. Proceedings of the International Conference on the Software Process (1996) 149-156
124. El Emam K., and Marshall P. Interrater agreement in assessment ratings. In: El Emam K., Drouin J.-N., and Melo W. (Eds). SPICE: Theory and Practice of Software Process Improvement and Capability Determination (1998), IEEE CS Press, New York
125. Hartmann D. Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis 10 1 (1977) 103-116
126. Suen H., and Lee P. The effects of the use of percentage agreement on behavioral observation reliabilities: a reassessment. Journal of Psychopathology and Behavioral Assessment 7 3 (1985) 221-234
127. Zwick R. Another look at interrater agreement. Psychological Bulletin 103 3 (1988) 374-378
128. Bennett E., Alpert R., and Goldstein A. Communications through limited response questioning. Public Opinion Quarterly 18 (1954) 303-308
129. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement XX 1 (1960) 37-46
130. Scott W. Reliability of content analysis: the case of nominal scale Coding. Public Opinion Quarterly 19 (1955) 321-325
131. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled agreement or partial credit. Psychological Bulletin 70 (1968) 213-220
132. Fleiss J. Statistical Methods for Rates and Proportions (1981), John Wiley, New York
133. No. 13, Fall. Khurana M., and El Emam K. Assessment experience and the reliability of assessments. Software Process Newsletter (1998), IEEE Technical Council on Software Engineering
134. Kerlinger F. Foundations of Behavioral Research (1986), Holt. Rinehart, and Winston, New York
135. Cronbach L. Test validation. In: Thorndike R. (Ed). Educational Measurement (1971), American Council on Education
136. El Emam K., and Birk A. Validating the ISO/IEC 15504 measures of software development process capability. Journal of Systems and Software 52 2 (2000) 119-149
137. Herbsleb J., Carleton A., Rozum J., et al. Benefits of CMM-based software process improvement: initial results. Technical Report CMU-SEI-94-TR-13 (1994), Software Engineering Institute
138. Dion R. Elements of a process improvement program. IEEE Software 9 4 (1992) 83-85
139. To appear. El Emam K., and Birk A. Validating the ISO/IEC 15504 measures of software requirements analysis process capability. IEEE Transactions on Software Engineering (2000)
140. Benno S., and Frailey D. Software process improvement in DSEG: 1989-1995. Texas Instruments Technical Journal 12 2 (1995) 20-28
141. Lipke W., and Butler K. Software process improvement: a success story. Crosstalk 5 9 (1992) 29-39
142. Butler K. The economic benefits of software process improvement. Crosstalk 8 7 (1995) 14-17
143. No. 5, Winter. Lebsanft L. Bootstrap: experiences with Europe's software process assessment and improvement method. available at. Software Process Newsletter (1996), IEEE Computer Society 6-10. http://www.se.cs.mcgill.ca/process/spn.html
144. Rubin H. Software process maturity: measuring its impact on productivity and quality. Proceedings of the International Conference on Software Engineering (1993) 468-476
145. Deephouse C., Goldenson D., Kellner M., and Mukhopadhyay T. The effects of software processes on meeting targets and quality. Proceedings of the Hawaiian International Conference on Systems Sciences, January vol. 4 (1995) 710-719
146. Krishnan M., and Kellner M. Measuring process consistency: implications for reducing software defects. IEEE Transactions on Software Engineering 25 6 (1999) 800-815
147. No. 5, Winter. Jones C. The pragmatics of software process improvements. available at. Software Process Newsletter (1996), IEEE Computer Society TCSE 1-4. http://www.seq.iit.nrc.ca/spn
148. Jones C. Assessment and Control of Software Risks (1994), Prentice-Hall, Englewood Cliffs, NJ
149. McGarry F., Burke S., and Decker B. Measuring the impacts individual process maturity attributes have on software projects. Proceedings of the 5th International Software Metrics Symposium (1998) 52-60
150. Flowe R., and Thordahl J. A correlational study of the SEI's capability maturity model and software development performance in DoD contracts. MSc thesis (1994), Air Force Institute of Technology
151. No. 7, Fall. Lawlis P., Flowe R., and Thordahl J. A correlational study of the CMM and software development performance. available at. Software Process Newsletter (1996), IEEE TCSE 1-5. www.seq.iit.nrc.ca/spn
152. El Emam K., and Briand L. Costs and benefits of software process improvement. In: Messnarz R., and Tully C. (Eds). Better Software Practice for Business Benefit: Principles and Experience (1999), IEEE CS Press, New York
153. Brodman J., and Johnson D. What small businesses and small organizations say about the CMM. Proceedings of the 16th International Conference on Software Engineering (1994) 331-340
154. Lee J., and Kim S. The relationship between procedural formalization in MIS development and MIS success. Information and Management 22 (1992) 89-111
155. Bach J. Enough about process: what we need are heroes. IEEE Software 12 2 (1995) 96-98
156. El Emam K., and Madhavji N.H. A method for instrumenting software evolution processes and an example application. Notes From The International Workshop on Software Evolution, Processes, and Measurements. Technical Report 94-04 NT (1994), Software Engineering Test Laboratory, Department of Computer Science, University of Idaho
157. El Emam, K. and Madhavji, N. H. (unpublished). The impact of user capability on requirements engineering success. Submitted for publication.
158. Curtis B., Krasner H., and Iscoe N. A field study of the software design process for large systems. Communications of the ACM 31 11 (1988) 1268-1286
159. El Emam K., and Madhavji N.H. A field study of requirements engineering practices in information systems development. Proceedings of the 2nd IEEE International Symposium on Requirements Engineering (1995) 68-80
160. Finlay P., and Mitchell A. Perceptions of the benefits from the introduction of CASF: an empirical study. MIS Quarterly December (1994) 353-370
161. Pfleeger S.-L., Fenton N., and Page S. Evaluating software engineering standards. IEEE Computer September (1994) 71-79
162. Pfleeger S.-L. The language of case studies and formal experiments. Software Engineering Notes October (1994) 16-20
163. Fenton N., Littlewood B., and Page S. Evaluating software engineering standards and methods. In: Thayer R., and McGettrick A. (Eds). Software Engineering: A European Perspective (1993), IEEE CS Press, New York
164. Fenton N., and Page S. Towards the evaluation of software engineering standards. Proceedings of the Software Engineering Standards Symposium (1993) 100-107
165. Fenton N., Pfleeger S.-L., Page S., and Thornton J. The SMARTIE standards evaluation methodology. Technical Report (1994), Centre for Software Reliability, City University, UK
166. Goldenson D.R., and El Emam K. The international SPICE trials: project description and initial results. Proceedings of the 8th Software Engineering Process Group Conference. May (1996)
167. Hollander M., and Wolfe D. Nonparametric Statistical Methods (1999), John Wiley, New York
168. Smith B., and El Emam K. Tansitioning to phase 2 of the SPICE trials. Proceedings of SPICE'96 (1996) 45-55
169. Lawlis P., Flowe R., and Thordahl J. A correlational study of the CMM and software development performance. Crosstalk September (1995) 21-25
170. Staff. A survey of the surveys on the Benefits of ISO 9000. 11. Software Process. Quality & ISO 9000 3 (November 1994) 1-5