Experimental methodologies for large-scale systems: A survey

Parallel Processing Letters

Volumn 19, Issue 3, 2009, Pages 399-418

  Gustedt, Jens ^a   Jeannot, Emmanuel ^a   Quinson, Martin ^a  

^a LORIA   (France)

Author keywords

Benchmarkisng; Emulation; Experimen talmethodologies; Experimental computer science; In situ; Simulation

Indexed keywords

BENCHMARKISNG; EMULATION; EXPERIMEN TALMETHODOLOGIES; EXPERIMENTAL COMPUTER SCIENCE; IN-SITU; SIMULATION;

COMPUTER SCIENCE; SOIL STRUCTURE INTERACTIONS;

EXPERIMENTS;

EID: 68749109676     PISSN: 01296264     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0129626409000304     Document Type: Conference Paper

References (43)

1. Peter J. Denning, D. E. Comer, David Gries, Michael C. Mulder, Allen Tucker, A. Joe Turner, and Paul R. Young. Computing as a discipline. Commun. ACM, 32(1):9-23, 1989.
2. Peter J. Denning. ACM President's Letter: What is experimental computer science? Commun. ACM, 23(10):543-544, 1980.
3. J. S. Marron, F. Herńandez-Campos, and F. D. Smith. Mice and elephants visualization of internet traffic. In W. Ḧardle and B. R̈onz, editors, Proceedings of 15th Conference on Computational Statistics. Physika Verlag, Heidelberg, August 2002.
4. Paul Luckowicz, Walter F. Tichy, Ernst A. Heinz, and Lutz Prechelet. Experimental evaluation in computer science: a quantitative study. Technical Report iratr-1994-17, University of Karlshrue, Germany, 1994.
5. Walter F. Tichy. Should Computer Scientists Experiment More? Computer, 31(5):32-40, 1998.
6. M.V. Zelkowitz and D.R. Wallace. Experimental models for validating technology. Computer, 31(5):23-31, May 1998.
7. Peter J. Denning. Is computer science science? Commun. ACM, 48(4):27-31, 2005.
8. Peter J. Denning. ACM president's letter: performance analysis: experimental computer science as its best. Commun. ACM, 24(11):725-727, 1981.
9. Dror G. Feitelson. Experimental Computer Science: The Need for a Cultural Change. Internet version: http://www.cs.huji.ac.il/∼feit/papers/exp05.pdf, December 2006.
10. D. D.Johnson. A theoretician's guide to the experimental analysis of algorithms, 2001. AT&T Labs Research. Available from http://www.research. att.com/∼dsj/papers/experguide.ps.
11. Algorille Team, Algorithms for the Grid. INRIA Research proposal, July 2006. Available at http://www.loria.fr/equipes/algorille/algorille2.pdf.
12. Franck Cappello and Daniel Etiemble. MPI versus MPI+OpenMP on IBM SP for the NAS benchmarks. In Proceedings of the 2000 ACM/IEEE conference on Super-computing, 2000.
13. Dan Tsafrir and Dror G. Feitelson. Instability in parallel job scheduling simulation: the role of workload flurries. In 20th International Parallel and Distributed Processing Symposium (IPDPS 2006), Rhodes Island,Greece, April 2006.
14. Vern Paxson and Sally Floyd. Wide-Area Traffic: The Failure of Poisson Modeling. IEEE/ACM Transactions on Networking, 3(3):226-224, June 1995.
15. The DAS-3 project: http://www.starplane.org/das3/.
16. The Grid 5000 project: http://www.grid5000.org/.
17. R. Bolze, F. Cappello, E. Caron, M. Dayd́e, F. Desprez, E. Jeannot, Y. J́egou, S. Lanteri, J. Leduc, N. Melab, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and I. Touche. Grid'5000: A Large Scale And Highly Re-configurable Experimental Grid Testbed. International Journal of High Performance Computing Applications, 20(4):481-494, November 2006.
18. Planet lab: http://www.planet-lab.org/.
19. The xen virtual machine monitor: http://www.cl.cam.ac.uk/research/srg/ netos/xen/.
20. Qemu: open source processor emulator http://www.qemu.org/.
21. Huaxia Xia, Holly Dail, Henri Casanova, and Andrew A. Chien. The MicroGrid: Using Online Simulation to Predict Application Performance in Diverse Grid Network Environments. In CLADE, page 52, 2004.
22. L.-C. Canon and E. Jeannot. Wrekavoc a Tool for Emulating Heterogeneity. In 15th IEEE Heterogeneous Computing Workshop (HCW'06), Island of Rhodes, Greece, April 2006.
23. D.H. Bailey, E. Barszcz, J.T. Barton, D.S. Browning, R.L. Carter, L. Dagum, R.A. Fatoohi, P.O. Frederickson, T.A. Lasinski, R.S. Schreiber, H.D. Simon, V. Venkatakrishnan, and S.K. Weeratunga. The Nas Parallel Benchmarks. International Journal of High Performance Computing Applications, 5(3):63-73, 1991.
24. Ewa Deelman, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Sonal Patil, Mei-Hui Su, Karan Vahi, and Miron Livny. Pegasus: Mapping scientific workflows onto the grid. In European Across Grids Conference, pages 11-20, 2004.
25. Mendel Rosenblum, Edouard Bugnion, Scott Devine, and Stephen A. Herrod. Using the simos machine simulator to study complex computer systems. In ACM Transactions on Modeling and Computer Simulation, volume 7, pages 78 - 103, New York, NY, USA, 1997. ACM.
26. George F. Riley. The Georgia Tech Network Simulator. In ACM SIGCOMM workshop on Models, Methods and Tools for Reproducible Network Research, pages 5-12, 2003.
27. Rajkumar Buyya and M. Manzur Murshed. GridSim: A Toolkit for the Modeling and Simulation of Distributed Resource Management and Scheduling for Grid Computing. CoRR, CS.DC/0203019, 2002.
28. Henri Casanova, Arnaud Legrand, and Martin Quinson. SimGrid: a Generic Framework for Large-Scale Distributed Experiments. In 10th IEEE International Conference on Computer Modeling and Simulation, March 2008.
29. Elliot Jaffe, Danny Bickson, and Scott Kirkpatrick. Everlab: a production platform for research in network experimentation and computation. In LISA'07: Proceedings of the 21st conference on Large Installation System Administration Conference, pages 1-11, Berkeley, CA, USA, 2007. USENIX Association.
30. The RAMP project: Research Accelerator for Multiple Processors.
31. Amin Vahdat, Ken Yocum, Kevin Walsh, Priya Mahadevan, Dejan Kostíc, Jeff Chase, and David Becker. Scalability and accuracy in a large-scale network emulator. SIGOPS Oper. Syst. Rev., 36(SI):271-284, 2002.
32. Brian White, Jay Lepreau, Leigh Stoller, Robert Ricci, Shashi Guruprasad, Mac New-bold, Mike Hibler, Chad Barb, and Abhijeet Joglekar. An integrated experimental environment for distributed systems and networks. In Proc. of the Fifth Symposium on Operating Systems Design and Implementation, pages 255-270, Boston, MA, December 2002. USENIX Association.
33. iproute2+tc notes: http://snafu.freedom.org/linux2.2/iproute-notes.html.
34. Luigi Rizzo. Dummynet: a simple approach to the evaluation of network protocols. SIGCOMM Comput. Commun. Rev., 27(1):31-41, 1997.
35. Jürgen Hofer and Thomas Fahringer. The Otho Toolkit - Synthesizing tailor-made scientific grid application wrapper services. Multiagent and Grid Systems, 3(3):281-298, 2007.
36. Anthony Sulistio, Chee Shin Yeo, and Rajkumar Buyya. A taxonomy of computer-based simulations and its mapping to parallel and distributed systems simulation tools. Softw., Pract. Exper., 34(7):653-673, 2004.
37. James H. Cowie, David M. Nicol, and Andy T. Ogielski. Modeling the global internet. Computing in Science and Engineering, 1(1):42-50, 1999.
38. Stephen Naicken, Anirban Basu, Barnaby Livingston, and Sethalat Rodhetbhai. Towards yet another peer-to-peer simulator. In Proc. Fourth International Working Conference Performance Modelling and Evaluation of Heterogeneous Networks (HET-NETs '06), September 2006.
39. Pedro Garćia, Carles Pairot, Rub́en Mond́ejar, Jordi Pujol, Helio Tejedor, and Robert Rallo. Planetsim: A new overlay network simulation framework. In Software Engineering and Middleware, SEM 2004, volume 3437 of LNCS, pages 123-137, Linz, Austria, March 2005.
40. Ḿark Jelasity, Alberto Montresor, Gian Paolo Jesi, and Spyros Voulgaris. PeerSim. http://peersim.sourceforge.net/.
41. William H. Bell, David G. Cameron, Luigi Capozza, A. Paul Millar, Kurt Stockinger, and Floriano Zini. OptorSim - A Grid Simulator for Studying Dynamic Data Replication Strategies. International J. of High Performance Computing Applications, 17(4), 2003.
42. Wim Depoorter, Nils Moor, Kurt Vanmechelen, and Jan Broeckhove. Scalability of grid simulators: An evaluation. In Euro-Par '08: Proceedings of the 14th international Euro-Par conference on Parallel Processing, pages 544-553, Berlin, Heidelberg, 2008. Springer-Verlag.
43. Kayo Fujiwara and Henri Casanova. Speed and Accuracy of Network Simulation in the SimGrid Framework. In Workshop on Network Simulation Tools (NSTools), 2007.