Wake up and smell the coffee: Evaluation methodology for the 21st century

Communications of the ACM

Volumn 51, Issue 8, 2008, Pages 83-89

  Blackburn, Stephen M ^a   McKinley, Kathryn S ^b   Garner, Robin ^a   Hoffmann, Chris ^c   Khan, Asjad M ^c   Bentzur, Rotem ^d   Diwan, Amer ^e   Feinberg, Daniel ^d   Frampton, Daniel ^a   Guyer, Samuel Z ^f   Hirzel, Martin ^g   Hosking, Antony ^h   Jump, Maria ^b   Lee, Han ⁱ   Moss, J Eliot B ^c   Phansalkar, Aashish ^b   Stefanović, Darko ^d   VanDrunen, Thomas ^j   Von Dincklage, Daniel ^e   Wiedermann, Ben ^b  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^c UNIVERSITY OF MASSACHUSETTS   (United States)

^d UNIVERSITY OF NEW MEXICO   (United States)

^e UNIVERSITY OF COLORADO   (United States)

^f TUFTS UNIVERSITY   (United States)

^g IBM   (United States)

^h PURDUE UNIVERSITY   (United States)

ⁱ INTEL CORPORATION   (United States)

^j WHEATON COLLEGE   (United States)

more..

Author keywords

[No Author keywords available]

Indexed keywords

21ST CENTURY; DIVERSE APPLICATIONS; EVALUATION METHODOLOGIES; EXPERIMENTAL DESIGN; FIRST ORDERS; JAVA APPLICATIONS; MULTI-CORE ARCHITECTURE; NON-DETERMINISM; RIGOROUS ANALYSIS; RUN-TIME SYSTEMS;

BENCHMARKING; CHLORINE COMPOUNDS; COMPUTER PROGRAMMING LANGUAGES; COMPUTER SCIENCE; CORUNDUM; INNOVATION; MACHINE DESIGN; STATISTICS; TECHNOLOGY;

JAVA PROGRAMMING LANGUAGE;

EID: 49249108501     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/1378704.1378723     Document Type: Article

References (18)

1. Adamson, A., Dagastine, D., and Sarne, S. SPECjbb2005 - A year in the life of a benchmark. 2007 SPEC Benchmark Workshop. SPEC, Jan. 2007.
2. Blackburn, S.M., Cheng P., and McKinley, K.S. Myths and realities: The performance impact of garbage collection. Proceedings of the ACM Conference on Measurement and Modeling Computer Systems, pp. 25-36, New York, NY, June 2004.
3. Blackburn, S.M., Garner, R., Hoffman, C., Khan, A.M., McKinley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S.Z., Hirzel, M., Hosking, A., Jump, M., Lee, H., Moss, J.E.B., Phansalkar, A., Stefanović, D., VanDrunen, T., von Dincklage, D., and Wiedermann, B. The DaCapo benchmarks: Java benchmarking development and analysis (extended version). Technical Report TR-CS-06-01, Dept. of Computer Science, Australian National University, 2006. http://www.dacapobench.org.
4. Blackburn, S.M., Garner, R., Hoffman, C., Khan, A.M., McKinley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S.Z., Hirzel, M., Hosking, A., Jump, M., Lee, H., Moss, J.E.B., Phansalkar, A., Stefanović, D., VanDrunen, T., von Dincklage, D.,and Wiedermann, B. The DaCapo benchmarks: Java benchmarking development and analysis. ACM Conference on Object-Oriented Programming Systems, Languages, and Applications, pp. 169-190, Oct. 2006.
5. Bodden, E., Hendren, L., and Lhoták, O. A staged static program analysis to improve the performance of runtime monitoring. 21st European Conference on Object-Oriented Programming, July 30th-August 3rd 2007, Berlin, Germany, number 4609 in Lecture Notes in Computer Science, pp. 525-549, Springer Verlag, 2007.
6. Chidamber, S.R. and Kemerer, C.F. A metrics suite for object-oriented design. IEEE Transactions on Software Engineering, 20(6):476-493, 1994.
7. Dunteman, G.H. Principal Components Analysis. Sage Publications, Newbury Park, CA, USA, 1989.
8. Georges, A., Buytaert, D., and Eeckhout, L. Statistically rigorous Java performance evaluation. ACM Conference on Object-Oriented Programming Systems, Languages, and Applications, pp. 57-76, Montreal, Quebec, Canada, 2007.
9. Huang, X., Blackburn, S.M., McKinley, K.S., Moss, J.E.B., Wang Z., and Cheng P. The garbage collection advantage: Improving mutator locality. ACM Conference on Object-Oriented Programming Systems, Languages, and Applications, pp. 69-80, Vancouver, BC, 2004.
10. Leyland, B. Auckland central business district supply failure. Power Engineering Journal, 12(3):109-114, 1998.
11. National Transportation Safety Board. NTSB urges bridge owners to perform load capacity calculations before modifications; I-35W investigation continues. SB-08-02. http://www.ntsb.gov/Pressrel/2008/080115.html, Jan. 2008.
12. Neelakantam, N., Rajwar, R., Srinivas, S., Srinivasan, U., and Zilles, C. Hardware atomicity for reliable software speculation. ACM/IEEE International Symposium on Computer Architecture, pp. 174-185, ACM, New York, NY, USA, 2007.
13. Perez, D.G., Mouchard, G., and Temam, O. MicroLib: A case for the quantitative comparison of micro-architecture mechanisms. International Symposium on Microarchitecture, pp. 43-54, Portland, OR, Dec. 2004.
14. Standard Performance Evaluation Corporation. SPECjvm98 Documentation, release 1.03 edition, March 1999.
15. Standard Performance Evaluation Corporation. SPECjbb2000 (Java Business Benchmark) Documentation, release 1.01 edition, 2001.
16. Stefanović, D. Properties of Age-Based Automatic Memory Reclamation Algorithms. PhD thesis, Department of Computer Science, University of Massachusetts, Amherst, Massachusetts, Dec. 1998.
17. Yi, J.J., Vandierendonck, H., Eeckhout, L., and Lilja, D.J. The exigency of benchmark and compiler drift: Designing tomorrow's processors with yesterday's tools. International Conference on Supercomputing, pp. 75-86, Cairns, Queensland, Australia, July 2006.
18. Yoo, R.M., Lee, H.-H. S., Lee, H., and K. Chow. Hierarchical means: Single number benchmarking with workload cluster analysis. IISWC 2007. IEEE 10th International Symposium on Workload Characterization, pp. 204-213, IEEE, 2007.