DoKnowMe: Towards a domain knowledge-driven methodology for performance evaluation

Performance Evaluation Review

Volumn 43, Issue 4, 2016, Pages 23-32

  Li, Zheng ^a   O'Brien, Liam ^b   Kihl, Maria ^a  

^a LUND UNIVERSITY   (Sweden)

^b GEOSCIENCE AUSTRALIA   (Australia)

Author keywords

Domain specific knowledge; Experimental computer science; Methodology; Performance evaluation

Indexed keywords

COMPUTATION THEORY; COMPUTER SOFTWARE SELECTION AND EVALUATION; OBJECT ORIENTED PROGRAMMING; QUALITY ASSURANCE;

DOMAIN-SPECIFIC KNOWLEDGE; EVALUATION DOMAINS; EVALUATION METHODOLOGIES; EVALUATION STRATEGIES; METHODOLOGY; PERFORMANCE EVALUATIONS; SOFTWARE QUALITY ASSURANCE; VALIDATION RESULTS;

QUALITY CONTROL;

EID: 85009754937     PISSN: 01635999     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2897356.2897360     Document Type: Conference Paper

References (23)

1. D. J. Bowen, M. Kreuter, B. Spring, L. Cofta-Woerpel, L. Linnan, D. Weiner, S. Bakken, C. P. Kaplan, L. Squiers, C. Fabrizio, and M. Fernandez. How we design feasibility studies. Am. J. Preventive Med., 36(5):452-457, May 2009.
2. P. Checkland and J. Scholes. Soft Systems Methodology in Action. John Wiley & Sons Ltd., New York, NY, September 1999.
3. P. J. Denning. Performance evaluation: Experimental computer science at its best. ACM SIGMETRICS Perform. Eval. Rev., 10(3):106-109, Fall 1981.
4. D. G. Feitelson. Experimental computer science: The need for a cultural change. http://www.cs.huji.ac.il/~feit/papers/exp05.pdf, 3 December 2006.
5. D. G. Feitelson. From repeatability to reproducibility and corroboration. ACM SIGOPS Operating Syst. Rev., 49(1):3-11, January 2015.
6. P. J. Fortier and H. E. Michel. Computer Systems Performance Evaluation and Prediction. Digital Press, Burlington, MA, July 2003.
7. J. Grossklags. Experimental economics and experimental computer science: A survey. In Proc. ExpCS 2007, pages 1-11, San Diego, CA, USA, 13-14 June 2007. ACM Press.
8. G. Heiser. Systems benchmarking crimes. http://www.cse.unsw.edu.au/~gernot/benchmarking-crimes.html, 13 March 2015.
9. S. Islam, K. Lee, A. Fekete, and A. Liu. How a consumer can measure elasticity for Cloud platforms. In Proc. ICPE 2012, pages 85-96, Boston, USA, 22-25 April 2012. ACM Press.
10. R. K. Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley Computer Publishing, John Wiley & Sons, Inc., New York, NY, April 1991.
11. Jean-Yves Le Boudec. Performance Evaluation of Computer and Communication Systems. EFPL Press, Lausanne, Switzerland, February 2011.
12. H. Koziolek. Performance evaluation of component-based software system: A survey. Perform. Eval., 67(8):634658, August 2010.
13. Z. Li, L. O'Brien, R. Cai, and H. Zhang. Boosting metrics for Cloud services evaluation - the last mile of using benchmark suites. In Proc. AINA 2013, pages 381-388, Barcelona, Spain, 25-28 March 2013. IEEE Computer Society.
14. Z. Li, L. O'Brien, H. Zhang, and R. Cai. A factor framework for experimental design for performance evaluation of commercial Cloud services. In Proc. CloudCom 2012, pages 169-176, Taipei, Taiwan, 3-6 December 2012. IEEE Computer Society.
15. Z. Li, L. O'Brien, H. Zhang, and R. Cai. On a catalogue of metrics for evaluating commercial Cloud services. In Proc. GRID 2012, pages 164-173, Beijing, China, 20-23 September 2012. IEEE Computer Society.
16. Z. Li, L. O'Brien, H. Zhang, and R. Cai. On the conceptualization of performance evaluation of IaaS services. IEEE Trans. Serv. Comput., 7(4):628-641, October-December 2014.
17. Z. Li, H. Zhang, L. O'Brien, R. Cai, and S. Flint. On evaluating commercial Cloud services: A systematic review. J. Syst. Softw., 86(9):2371-2393, September 2013.
18. D. C. Montgomery. Design and Analysis of Experiments. John Wiley & Sons, Inc., Hoboken, NJ, 7th edition edition, January 2009.
19. M. S. Obaidat and N. A. Boudriga. Fundamentals of Performance Evaluation of Computer and Telecommunication Systems. John Wiley & Sons, Inc., Hoboken, New Jersey, January 2010.
20. R. Osman and W. J. Knottenbelt. Database system performance evaluation models: A survey. Perform. Eval., 69(10):471-493, October 2012.
21. B. A. Price, R. M. Baecker, and I. S. Small. A principled taxonomy of software visualization. J. Visual Lang. Comput., 4(3):211-266, September 1993.
22. P. Runeson and M. Höst. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng., 14(2):131-164, April 2009.
23. J. v. Kistowski, N. Herbst, and S. Kounev. Modeling variations in load intensity over time. In Proc. LT 2014, pages 1-4, Dublin, Ireland, 22 March 2014. ACM Press.