Automatic comparison of load tests to support the performance analysis of large enterprise systems

Proceedings of the European Conference on Software Maintenance and Reengineering, CSMR

Volumn , Issue , 2010, Pages 222-231

  Malik, Haroon ^a   Jiang, Zhen Ming ^a   Adams, Bram ^a   Hassan, Ahmed E ^a   Flora, Parminder ^b   Hamann, Gilbert ^b  

^a QUEEN S UNIVERSITY   (Canada)

^b RESEARCH IN MOTION   (Canada)

Author keywords

Load test; PCA; Performance counters

Indexed keywords

LARGE SCALE SYSTEMS; LOAD TESTING; PROGRAM DEBUGGING;

LARGE ENTERPRISE; PERFORMANCE ANALYSIS; PERFORMANCE BUGS; PERFORMANCE COUNTERS; PERFORMANCE DATA; PERFORMANCE GAIN; RESOURCE USAGE; STATISTICAL TECHNIQUES;

PRINCIPAL COMPONENT ANALYSIS;

EID: 79952632150     PISSN: 15345351     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CSMR.2010.39     Document Type: Conference Paper

References (37)

1. Avritzer. A., Larson. B., "Load testing software using deterministic state testing", In Proceedings of the ACM SIGSOFT international symposium on Software, 1993.
2. Beizer. B., "Software System Testing and Quality Assurance"' Van Nostrand Reinhold, March 1984.
3. Box. M. J., Box. R. M., "Computation of the Variance Ratio Distribution", The Computer Journal, pp. 277-278, 1969.
4. Brace, N., Kemp, R., Snelgar, R., "SPSS for Psychologists: Palgrave Macmillan", 2003.
5. Bruce Thompson., "Exploratory and Confirmatory Factor Analysis: Understanding Concepts and Applications", ISBN: 1-59147-093-5.
6. Chatterjee, C., Roychowdhury, V.P., Ramos, J., and Zoltowski, M.D., "Self-organizing algorithms for generalized eigen-decomposition," Neural Networks IEEE Transactions, vol.8, no.6, pp.1518-1530, Nov 1997. (Pubitemid 127765179)
7. Cohen, I., Zhang, S.,. Goldszmidt, M., Symons, J., Kelly, T., "Capturing, indexing, clustering, and retrieving system history", In Proceedings of the twentieth ACM symposium on Operating systems principles, 2005.
8. Hair, J.F., Jr.l Anderson, R. E.; Tatham, R. L.; and Black, W. C. (1998). "Multivariate data analysis with readings", 5th ed.. Englewood Cliffs, NJ: Prentice-Hall.
9. Huck, K.A.; Malony, A.D., "PerfExplorer: A Performance Data Mining Framework For Large-Scale Parallel Computing," Supercomputing, 2005. Proceedings of the ACM/IEEE SC 2005 Conference, vol., no., pp. 41-41, 2005.
10. Jiang, Z. M., Hassan, A. E., "Automatic identification of load testing problems". In Proceedings of the 24th IEEE International Conference on Software Maintenance (ICSM), 2008.
11. Jiang, Z. M., Hassan, A. E., Hamann, G., Flora, P., Automated Performance Analysis of Load Tests. In Proceedings of the 25th IEEE International Conference on Software Maintenance (ICSM) 2009, Edmonton, Canada, September 20-26, 2009.
12. Jolliffe IT., "Principal Component Analysis", Second Edition. New York, Springer-Verlag; (Springer Series in Statistics), 2002.
13. Kaiser, H. F., "An Index of Factorial Simplicity," Psychometrika, vol. 39, pp. 31-36, 1974.
14. Lendasse, A., Lee, J., Bodt, E.d., Wertz, V. and Verleyen, M., "Input Data Reduction for the Prediction of Financial Time series", 2000.
15. M.W. Knop, J.M. Schopf and P.A. Dinda, "Windows performance monitoring and data reduction using watch tower", Workshop on Self-Healing, Adaptive and self-MANaged Systems (SHAMAN), 2002.
16. May, J. M., "Software for multiplexing hardware performance counters in multithreaded programs". Proceedings of 2001 International Parallel and Distributed Processing Symposium, April 2001.
17. Mendes, C.L. and Reed, D.A., "Monitoring Larger Systems Via Statistical Sampling", Proceedings of the LACSI Symposium, Santa Fe, 2002.
18. Montanelli, R.G. Jr., and Humphreys, L. G., "Latent roots of random data correlation matrices with squared multiple correlations on the diagonal: A monte carlo study", Psychometrika, pp. 341-348, 1976.
19. Nickolayev, O.Y., Roth, P.C. and Reed, D.A., "Real-Time Statistical Clustering For Event Trace Reduction", The International Journal of Supercomputer Applications and High Performance Computing, pp 144-159, 1997.
20. Parets, J., and Torres, J.C., "Software Maintenance versus Software Evolution: An Approach to Software Systems Evolution", Workshop on Engineering of Computer Based Systems (ECBS'96), pp. 134, 1996.
21. Qu, Y., Adam, B.l., Thornquist, M., Potter, J.D., Thompson, M.L., Yasui, Y., Davis, J., Schellhammer, P., Cazares, L., Clements, M., Jr., G.L.W. and Feng, Z., "Data Reduction Using a Discrete Wavelet Transform in Discriminant Analysis of Very High Dimensionality Data", Biometrics, 2003.
22. Raubenheimer, J. E., "An item selection procedure to maximize scale reliability and validity". South African Journal of Industrial Psychology, 30 (4), 59-64, 2004.
23. Reed, D.A., Aydt, R.A., Noe, R.J., Roth, P.C., Shields, K.A., Schwartz, B.W. and Travera, L.F., "Scalable Performance Analysis: The pablo Performance Analysis Environment". Proceedings of the Scalable Parallel Libraries Conference, 1993.
24. Ringberg, H., Soule, A., Rexford, J., Diot, C., "Sensitivity of PCA for traffic anomaly detection", In ACM SIGMETRICS,San Diego, CA, USA, 2007.
25. Sandeep, S. Ratna., Swapna, M, Thirumale Niranjan., Sai Susarla., Siddhartha Nandi., "CLUEBOX: A Performance Log Analyzer for Automated Troubleshooting"' WASL, 2008.
26. Schwartz. Jeffry. A., "Utilizing performance monitor counters to effectively guide windows and SQL server tuning efforts"', pp. 933-944, 2006.
27. Sébastien Lê., Julie J., François Husson., "FactoMineR: An R Package for Multivariate Analysis", Journal of Statistical Software, pp.1-18, 2008.
28. Sprunt, B., "The basics of performance-monitoring hardware," Micro, IEEE, vol.22, no.4, pp. 64-71, Jul 2002. (Pubitemid 34979926)
29. Thakkar, D., Hassan, A.E Hamann, G., Flora, P., "A framework for measurement based performance modeling". In WOSP '08: Proceedings of the 7th international workshop on Software and performance, pages 55-66, New York, NY, USA, 2008.
30. The R Foundation for Statistical Computing., "R Project for Statistical Computing"' http://www.r-project.org, 2007.
31. Yoccoz, N.G., J. D. Nichols and Boulinier, T., "Monitoring of Biological Diversity in space and time, Trends in Ecology and Evolution", 2001.
32. Rencher, A. C. Methods of Multivariate Analysis. Wiley, New York, 1995.
33. K. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft. When is "nearest neighbor" meaningful? In Lecture Notes in Computer Science, volume 1540, pages 217-235, 1999.
34. A. Hinneburg, C. Aggarwal, and D. Keim. What is the nearest neighbor in high dimensional spaces? In The VLDB Journal, pages 506-515, 2000.
35. B. Milenova and M. Campos. O-cluster: Scalable clustering of large high dimensional data sets. Oracle Corporation, 2002.
36. Limitations of Maximum Likelihood Estimation Procedures When a Majority of the Observations Are Below the Limit of Detection Ram B. Jain, Richard Y. Wang Analytical Chemistry 2008 80 (12), 4767-4772 (Pubitemid 351842358)
37. Ahn. D., and J. Vetter., 'Scalable analysis techniques for microprocessor performance counter metrics'. In Proceedings of Supercomputing, 2002.