Evaluating the relative performance of engineering design projects: A case study using data envelopment analysis

IEEE Transactions on Engineering Management

Volumn 53, Issue 3, 2006, Pages 471-482

  Farris, Jennifer A ^a   Groesbeck, Richard L ^a   Van Aken, Eileen M ^a   Letens, Geert ^a  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

Author keywords

Concurrent engineering; ISO 15288; Military organizations; Performance measurement; Project management

Indexed keywords

DATA REDUCTION; DECISION MAKING; EVALUATION; PROJECT MANAGEMENT; SOCIETIES AND INSTITUTIONS;

DATA ENVELOPMENT ANALYSIS; ENGINEERING DESIGN PROJECTS; PROJECT EVALUATION TOOL; PROJECT-BASED ORGANIZATIONS;

SYSTEMS ENGINEERING;

EID: 33746627741     PISSN: 00189391     EISSN: None     Source Type: Journal    
DOI: 10.1109/TEM.2006.878100     Document Type: Article

References (84)

1. S. Newell, "Enhancing cross-project learning," Eng. Manag. J., vol. 16, no. 1, pp. 12-20, Mar. 2004.
2. A Guide to the Project Management Body of Knowledge Project Management Institute, Newton Square, PA, 2004, ANSI/PMI 99-001-2004, 3rd.
3. T. Cooke-Davies, "The 'real' success factors on projects," Int. J. Project Manag., vol. 20, no. 3, pp. 185-190, Apr. 2002.
4. J. P. Lewis, The Project Manager's Desk Reference. New York:MC-Graw Hill, 2000.
5. "Effective Benchmarking for Project Management," White Paper Project Management Institute, Newton Square, PA, 2004.
6. A Practice Standard for Earned Value Management Project Management Institute, Newton Square, PA, 2005.
7. A. J. Shenhar, A. Tishler, D. Dvir, S. Lipovetsky, and T. Lechler, "Refining the search for project success factors: A multivariate, typological approach," R & D Manag., vol. 32, no. 2, pp. 111-126, Mar. 2002.
8. R. J. Might and W. A. Fischer, "The role of structural factors in determining project management success," IEEE Trans. Eng. Manag., vol. EM-32, pp. 71-77, May 1985.
9. J. K. Pinto and D. P. Slevin, "Project success: Definitions and measurement techniques," Project Manag. J., vol. 19, no. 1, pp. 67-73, Feb. 1988.
10. A. M. M. Liu and A. Walker, "Evaluation of project outcomes," Construction Manag. Econ., vol. 16, no. 2, pp. 209-219, Mar. 1998.
11. D. Dvir, S. Lipovetsky, A. Shenhar, and A. Tishler, "In search of project classification: A non-universal approach to project success factors," Res. Policy, vol. 27, pp. 915-935, 1998.
12. H. Kerzner, "In search of excellence in project management," J. Syst. Manag., vol. 38, no. 2, pp. 30-39, Feb. 1987.
13. M. E. Pate-Cornell and R. L. Dillon, "Success factors and future challenges in the management of faster-better-cheaper projects: Lessons learned from NASA," IEEE Trans. Eng. Manag., vol. 48, no. 1, pp. 25-35, Feb. 2001.
14. J. K. Pinto and S. J. Mantel Jr., "The causes of project failure," IEEE Trans. Eng. Manag., vol. 37, no. 4, pp. 269-275, Nov. 1990.
15. J. K. Pinto and D. P. Slevin, "Critical factors in successful project implementation," IEEE Trans. Eng. Manag., vol. EM-34, no. 1, pp. 22-27, Feb. 1987.
16. J. K. Pinto and D. P. Slevin, "Critical success factors in R&D projects," Res. Technol. Manag., vol. 32, no. 1, pp. 31-35, Jan./Feb. 1989.
17. W. Belassi and O. I. Tukel, "A new framework for determining critical success/failure factors in projects," Int. J. Project Manag., vol. 14, no. 3, pp. 141-151, Jun. 1996.
18. M. Freeman and P. Beale, "Measuring project success," Project Manag. J., vol. 23, no. 1, pp. 8-17, Mar. 1992.
19. S. W. Hughes, D. D. Tippett, and W. K. Thomas, "Measuring project success in the construction industry," Eng. Manag. J., vol. 16, no. 3, pp. 31-37, Sep. 2004.
20. A. S. Pillai, A. Joslu, and K. S. Rao, "Performance measurement of R&D projects in a multi-project, concurrent engineering environment," Int. J. Project Manag., vol. 20, no. 2, pp. 165-177, Feb. 2002.
21. D. P. Slevin and J. K. Pinto, "The project implementation profile: New tool for project managers," Project Manag. J., vol. 17, no. 4, pp. 57-70, Sep. 1986.
22. E. S. Andersen and S. A. Jessen, "Project evaluation scheme: A tool for evaluating project status and predicting results," Project Manag., vol. 6, no. 1, pp. 61-69, 2000.
23. A. Charnes, W. W. Cooper, and E. Rhodes, "Measuring the efficiency of decision making units," Eur. J. Oper. Res., vol. 2, no. 6, pp. 429-444, Nov. 1978.
24. A. Charnes, W. W. Cooper, A. Y. Lewin, and L. M. Seiford, Eds., Data Envelopment Analysis: Theory, Methodology, and Application. Boston, MA: Kluwer Academic, 1994.
25. E. Thanassoulis, Introduction to the Theory and Application of Data Envelopment Analysis. Boston, MA: Kluwer Academic, 2001.
26. B. Hollingsworth, "A review of data envelopment analysis software," Econ. J., vol. 107, no. 443, pp. 1268-1270, Jul. 1997.
27. I. Herrero and S. Pascoe, Estimation of Technical Efficiency: A Review of Some of the Stochastic Frontier and DEA Software CHEER, 2002, vol. 15(1) [Online]. Available: http://www.economicsnetwork.ac.uk/cheer/ ch15_1/dea.htm
28. Frontier Analyst®Professional, 3.0.3 ed. Kendal, Cumbria, UK, Banxia Software Ltd., 2001.
29. M. A. Mahmood, K. J. Pettingell, and A. I. Shaskevich, "Measuring productivity of software projects: A data envelopment analysis approach," Decision Sci.., vol. 27, no. 1, pp. 57-80, Winter 1996.
30. P. D. Chatzoglou and A. C. Soteriou, "A DEA framework to assess the efficiency of the software requirements capture and analysis process," Decision Sci.., vol. 30, no. 2, pp. 503-531, Spring 1999.
31. P. Kauffmann, R. Unal, A. Fernandez, and C. Keating, "A model for allocating resources to research programs by evaluating technical importance and research productivity," Eng. Manag. J., vol. 12, no. 1, pp. 5-8, Mar. 2000.
32. R. G. Dyson, R. Allen, A. S. Camanho, V. V. Podinovski, C. S. Sarrico, and E. A. Shale, "Pitfalls and protocols in DEA," Eur. J. Oper. Res., vol. 132, no. 2, pp. 245-259, Jul. 2001.
33. W. W. Cooper, L. M. Seiford, and K. Tone, Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software. Boston, MA: Kluwer Academic, 2000.
34. A. Charnes, C. T. Clark, W. W. Cooper, and B. Golany, "A developmental study of data envelopment analysis in measuring the efficiency of maintenance units in the US air forces," Ann. Operations Res., vol. 2, pp. 95-112, 1985.
35. Y. Roll, B. Golany, and D. Seroussy, "Measuring the efficiency of maintenance units in the Israeli air force," Eur. J. Oper. Res., vol. 27, no. 2, pp. 136-142, Nov. 1989.
36. R. L. Clarke, "Evaluating USAF vehicle maintenance productivity over time: An application of data envelopment analysis," Decision Sci., vol. 23, no. 2, pp. 376-384, Mar./Apr. 1992.
37. S. Sun, "Assessing joint maintenance shops in the Taiwanese army using data envelopment analysis," J. Oper. Manag., vol. 22, no. 3, pp. 233-245, Jun. 2004.
38. J. C. Paradi, Smith, and Schaffnit-Chatterjee, "Knowledge worker performance analysis using DEA: An application to engineering design teams at Bell Canada," IEEE Trans. Eng. Manag., vol. 49, no. 2, pp. 161-172, May 2002.
39. R. D. Banker, S. M. Datar, and C. F. Kemerer, "Factors affecting software maintenance productivity: An exploratory study," in Proc. 8th Int. Conf. Information Systems, 1987, pp. 160-175.
40. R. D. Banker, S. M. Datar, and C. F. Kemerer, "A model to evaluate variables impacting the productivity of software maintenance projects," Manag. Sci., vol. 37, no. 1, pp. 1-18, Jan. 1991.
41. J. C. Paradi, D. N. Reese, and D. Rosen, "Applications of DEA to measure the efficiency of software production at two large Canadian banks," Ann. Oper. Res., vol. 73, pp. 91-115, 1997.
42. C. Parkan, K. Lam, and G. Hang, "Operational competitiveness analysis on software development," J. Oper. Res. Soc., vol. 48, no. 9, pp. 892-905, Sep. 1997.
43. R. D. Banker and C. F. Kemerer, "Scale Economics in New Software Development," IEEE Trans. Softw. Eng., vol. 15, pp. 1199-1205, Oct. 1989.
44. R. D. Banker, H. Chang, and C. F. Kemerer, "Evidence on economies of scale in software development," Inf. Softw. Technol., vol. 36, no. 15, pp. 275-282, May 1994.
45. R. D. Banker and S. A. Slaughter, "A field study of scale economies in software maintenance," Manag. Sci., vol. 43, no. 12, pp. 1709-1725, Dec. 1997.
46. E. Stensrud and I. Myriveit, "Identifying high performance ERP projects," IEEE Trans. Softw. Eng., vol. 29, no. 5, pp. 398-416, May 2003.
47. Z. Yang and J. C. Paradi, "DEA evaluation of a Y2K software retrofit program," IEEE Trans. Eng. Manag., vol. 51, no. 3, pp. 279-287, Aug. 2004.
48. M. Oral, O. Kettani, and P. Lang, "A methodology for collective evaluation and selection of industrial R&D projects," Manag. Sci., vol. 37, no. 7, pp. 871-885, Jul. 1991.
49. W. D. Cook, M. Kress, and L. M. Seiford, "Data envelopment analysis inn the presence of both quantitative and qualitative factors," J. Oper. Res. Soc., vol. 47, no. 7, pp. 945-953, Jul. 1996.
50. R. H. Green, J. R. Doyle, and W. D. Cook, "Preference voting and project ranking using DEA and cross-evaluation," Eur. J. Oper. Res., vol. 90, no. 3, pp. 461-472, May 1996.
51. J. D. Linton, S. T. Walsh, and J. Morabito, "Analysis, ranking and selection of R&D projects in a portfolio," R&D Manag., vol. 32, no. 2, pp. 139-147, Winter 2002.
52. S. A. Thore and L. Lapao,, S.A. Thore, Ed., "Prioritizing R&D projects in the face of technological and market uncertainty: Combining scenario analysis and DEA," in Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation. Boston, MA: Kluwer Academic, 2002, pp. 87-110.
53. S. A. Thore and G. Rich, S. A. Thore, Ed., "Prioritizing a portfolio of R&D activities, employing data envelopment analysis," in Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation. Boston, MA: Kluwer Academic, 2002, pp. 53-74.
54. C. C. Liu and C. Y. Chen, "A two-dimensional model for allocating resources to R&D programs," J. Amer. Arad. Bus., vol. 5, no. 1/2, pp. 469-473, Sep. 2004.
55. H. Eliat, B. Golany, and A. Shtub, "Constructing and evaluating balanced portfolios of R&D projects with interactions: A DEA based methodology," Eur. J. Oper. Res., 2005, to be published.
56. D. Verma and K. K. Sinha, "Toward a theory of project interdependencies in high tech R&D environments," J. Oper. Manag., vol. 20, no. 1, pp. 451-468, Sep. 2002.
57. B. Yuan and J. N. Huang, S. A. Thore, Ed., "Applying data envelopment analysis to evaluate the efficiency of R&D projects - A case study of R&D energy technology," in Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation. Boston, MA: Kluwer Academic, 2002, pp. 111-134.
58. E. Revilla, J. Sarkis, and A. Modrego, "Evaluating performance of public-private research collaborations," J. Oper. Res. Soc., vol. 54, no. 2, pp. 165-174, Feb. 2003.
59. D. K. Chai and D. C. Ho, "Multiple criteria decision model for resource allocation: A case study in an electric utility," INFOR, vol. 36, no. 3, pp. 151-160, Aug. 1998.
60. S. A. Thore and F. Pimentel, S. A. Thore, Ed., "Evaluating a portfolio of proposed projects, ranking the relative to a list of existing projects," in Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation. Boston, MA: Kluwer Academic, 2002, pp. 233-246.
61. J. D. Linton and W. D. Cook, "Technology implementation: A comparative study of Canadian and U.S. factories," INFOR, vol. 36, no. 3, pp. 142-150, Aug. 1998.
62. J. S. Busby and A. Williamson, "The appropriate use of performance measurement in non-production activities: The case of engineering design," Int. J. Operations Production Manag., vol. 20, no. 3, pp. 336-358, Mar. 2000.
63. E. Van Aken, D. Van Goubergen, and G. Letens, "Integrated enterprise transformation: Case application in a project organization in the Belgian armed forces," Eng. Manag. J., vol. 15, no. 2, pp. 3-16, Jun. 2003.
64. Systems Engineering - System Life Cycle Processes International Organization for Standardization, Geneva, 2002, ISO/IEC 15288: 2002.
65. D. Gerwin and N. J. Barrowman, "An evaluation of research on integrated product development," Manag. Sci., vol. 48, no. 7, pp. 938-953, Jul. 2002.
66. M. Ainscough and B. Yazdani, "Concurrent engineering within British industry," Concurrent Eng.: Res. Applicat., vol. 8, no. 1, pp. 2-11, Mar. 2000.
67. H. Maylor and R. Gosling, "The reality of concurrent new product development," Integrated Manuf. Syst., vol. 9, no. 2, pp. 69-76, 1998.
68. B. Haque, "Problems in concurrent product development: An in-depth comparative study of three companies," Integrated Manufact. Syst., vol. 14, no. 3, pp. 191-207, May 2003.
69. N. Bhuiyan, D. Gerwin, and V. Thomson, "Simulation of the new product development process for performance improvement," Manag. Sci., vol. 50, no. 12, pp. 1690-1703, Dec. 2004.
70. C. Terwiesch and C. H. Loch, "Measuring the effectiveness of overlapping development activities," Manag. Sci., vol. 45, no. 4, pp. 455-465, Apr. 1999.
71. K. Eisenhardt and B. Tabrizi, "Accelerating adaptive processes: Product innovation in the global computer industry," Admin. Sci. Quart., vol. 40, no. 1, pp. 84-110, Mar. 1995.
72. A. Griffin, "The effect of project and process characteristics on product development cycle time," J. Marketing Res., vol. 34, no. 1, pp. 24-35, Feb. 1997.
73. H. D. Sherman, R. H. Silkman,Ed., "Managing productivity of health care organizations," in Measuring Efficiency: An Assessment of Data Envelopment Analysis. San Francisco: Jossey-Bass, 1986, pp. 31-46.
74. H. Kerzner, In Search of Excellence in Project Management. New York: Van Nostrand Reinhold, 1998.
75. A. P. C. Chan and A. P. L. Chan, "Key performance indicators for measuring construction success," Benchmarking, vol. 11, no. 2, pp. 203-221, 2004.
76. O. I. Tukel and W. O. Rom, "An empirical investigation of project evaluation criteria," Int. J. Oper. Production Manag., vol. 21, no. 3, pp. 400-416, Mar. 2001.
77. A. J. Shenhar and D. Dvir, "Toward a typological theory of project management," Res. Policy, vol. 25, no. 4, pp. 607-632, Jun. 1996.
78. V. S. D. Krishnan, "Managing the simultaneous execution of coupled phases in concurrent product development," IEEE Trans. Eng. Manag., vol. 43, no. 3, pp. 210-217, May 1996.
79. T. Raz, A. J. Shenhar, and D. Dvir, "Risk management, project success, and technological uncertainty," R&D Manag., vol. 32, no. 2, pp. 101-109, Mar. 2002.
80. R. D. Banker, A. Charnes, and W. W. Cooper, "Some models for estimating technical and scale inefficiencies in data envelopment analysis," Manag. Sci., vol. 30, no. 9, pp. 1078-1092, Sep. 1984.
81. H. Scheel, "Undesirable outputs in efficiency valuations," Eur. J. Oper. Res., vol. 132, no. 2, pp. 400-410, Jul. 2001.
82. P. L. Brockett and B. Golany, "Using rank statistics for determining programmatic efficiency differences in data envelopment analysis," Manag. Sci., vol. 42, no. 3, pp. 466-472, Mar. 1996.
83. T. Sueyoshi and S. Aoki, "A use of a nonparametric statistic for DEA frontier shift: The Kruskal and Wallis rank test," Omega, vol. 29, no. 1, pp. 1-18, Feb. 2001.
84. B. Golany and S. A. Thore, S. A. Thore, Ed., "On the ranking of R&D projects in a hierarchical organizational structure subject to global resource constraints," in Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation. Boston, MA: Kluwer Academic, 2002, pp. 253-274.