Cloud service benchmarking: Measuring quality of cloud services from a client perspective

Cloud Service Benchmarking: Measuring Quality of Cloud Services from a Client Perspective

Volumn , Issue , 2017, Pages 1-167

  Bermbach, David ^a   Wittern, Erik ^b   Tai, Stefan ^a  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85020498620     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-319-55483-9     Document Type: Book

References (77)

1. Abadi, D.: Consistency tradeoffs in modern distributed database system design: Cap is only part of the story. IEEE Computer 45(2), 37-42 (2012). DOI 10.1109/MC.2012.33. URL http://dx.doi.org/10.1109/MC.2012.33
2. Anderson, E., Li, X., Shah, M.A., Tucek, J., Wylie, J.J.: What consistency does your keyvalue store actually provide? In: Proceedings of the 6th Workshop on Hot Topics in System Dependability (HOTDEP), HotDep'10, pp. 1-16. USENIX Association, Berkeley, CA, USA (2010). URL http://dl.acm.org/citation.cfm?id=1924908.1924919
3. Bailis, P., Ghodsi, A., Hellerstein, J.M., Stoica, I.: Bolt-on causal consistency. In: Proceedings of the 33rd International Conference on Management of Data (SIGMOD), SIGMOD '13, pp. 761-772. ACM, New York, NY, USA (2013). DOI 10.1145/2463676.2465279. URL http://doi.acm.org/10.1145/2463676.2465279
4. Bailis, P., Venkataraman, S., Franklin, M.J., Hellerstein, J.M., Stoica, I.: Probabilistically bounded staleness for practical partial quorums. Proceedings of the VLDB Endowment 5(8), 776-787 (2012). URL http://dl.acm.org/citation.cfm?id=2212351. 2212359
5. Bermbach, D.: Benchmarking eventually consistent distributed storage systems. Ph.D. thesis, Karlsruhe Institute of Technology (2014)
6. Bermbach, D.: Quality of cloud services: Expect the unexpected. IEEE Internet Computing (2017)
7. Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems: An overview of models, metrics and measurement approaches. In: Proceedings of the International Conference on Networked Systems (NETYS), pp. 175-189. Springer Berlin Heidelberg (2013)
8. Bermbach, D., Kuhlenkamp, J., Derre, B., Klems, M., Tai, S.: A middleware guaranteeing client-centric consistency on top of eventually consistent datastores. In: Proceedings of the 1st International Conference on Cloud Engineering (IC2E), IC2E '13, pp. 114-123. IEEE Computer Society, Washington, DC, USA (2013). DOI 10.1109/IC2E.2013.32. URL http: //dx.doi.org/10.1109/IC2E.2013.32
9. Bermbach, D., Kuhlenkamp, J., Dey, A., Sakr, S., Nambiar, R.: Towards an Extensible Middleware for Database Benchmarking. In: Proceedings of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC), pp. 82-96. Springer (2014)
10. Bermbach, D., Tai, S.: Eventual consistency: How soon is eventual? an evaluation of amazon s3's consistency behavior. In: Proceedings of the 6th Workshop on Middleware for Service Oriented Computing (MW4SOC), MW4SOC '11, pp. 1:1-1:6. ACM, New York, NY, USA (2011). DOI 10.1145/2093185.2093186. URL http://doi.acm.org/10.1145/ 2093185.2093186
11. Bermbach, D., Tai, S.: Benchmarking eventual consistency: Lessons learned from long-term experimental studies. In: Proceedings of the 2nd International Conference on Cloud Engineering (IC2E). IEEE (2014)
12. Bermbach, D., Wittern, E.: Benchmarking web api quality. In: Proceedings of the 16th International Conference on Web Engineering (ICWE 2016). Springer (2016)
13. Bermbach, D., Zhao, L., Sakr, S.: Towards comprehensive measurement of consistency guarantees for cloud-hosted data storage services. In: R. Nambiar, M. Poess (eds.) Performance Characterization and Benchmarking, Lecture Notes in Computer Science, vol. 8391, pp. 32-47. Springer International Publishing (2014). DOI 10.1007/978-3-319-04936-6 3. URL http://dx.doi.org/10.1007/978-3-319-04936-6_3
14. Binnig, C., Kossmann, D., Kraska, T., Loesing, S.: How is the weather tomorrow?: Towards a benchmark for the cloud. In: Proceedings of the 2nd International Workshop on Testing Database Systems (DBTEST), DBTest '09, pp. 9:1-9:6. ACM, New York, NY, USA (2009). DOI 10.1145/1594156.1594168. URL http://doi.acm.org/10.1145/1594156. 1594168
15. Borhani, A.H., Leitner, P., Lee, B.S., Li, X., Hung, T.: Wpress: An application-driven performance benchmark for cloud-based virtual machines. Proceedings of the 2014 IEEE 18th International Enterprise Distributed Object Computing Conference (EDOC) pp. 101-109 (2014)
16. Brewer, E.: Podc keynote. http://www.cs.berkeley.edu/brewer/ cs262b-2004/PODC-keynote.pdf (accessed Jun 27,2013) (2000)
17. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A distributed storage system for structured data. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), OSDI '06, pp. 205-218. USENIX Association, Berkeley, CA, USA (2006). URL http: //dl.acm.org/citation.cfm?id=1298455.1298475
18. Cooper, B.F., Ramakrishnan, R., Srivastava, U., Silberstein, A., Bohannon, P., Jacobsen, H.A., Puz, N., Weaver, D., Yerneni, R.: Pnuts: Yahoo!'s hosted data serving platform. Proceedings of the VLDB Endowment 1(2), 1277-1288 (2008). URL http://dl.acm.org/ citation.cfm?id=1454159.1454167
19. Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st Symposium on Cloud Computing (SOCC), SOCC '10, pp. 143-154. ACM, New York, NY, USA (2010). DOI 10.1145/1807128.1807152. URL http://doi.acm.org/10.1145/1807128.1807152
20. Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), OSDI'04, pp. 10-10. USENIX Association, Berkeley, CA, USA (2004). URL http: //dl.acm.org/citation.cfm?id=1251254.1251264
21. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon's highly available key-value store. In: Proceedings of 21st Symposium on Operating Systems Principles (SOSP), SOSP '07, pp. 205-220. ACM, New York, NY, USA (2007). DOI 10.1145/1294261.1294281. URL http://doi.acm.org/10.1145/1294261.1294281
22. Difallah, D.E., Pavlo, A., Curino, C., Cudre-Mauroux, P.: Oltp-bench: An extensible testbed for benchmarking relational databases. Proceedings of the VLDB Endowment 7(4), 277-288 (2013)
23. Fidge, C.J.: Timestamps in message-passing systems that preserve the partial ordering. In: Proceedings of the 11th Australian Computer Science Conference (ACSC), vol. 10, pp. 56-66 (1988)
24. Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the cloud: What it should, can, and cannot be. In: Proceedings of the 4th TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2012), pp. 173-188. Springer (2013)
25. Fox, A., Brewer, E.A.: Harvest, yield, and scalable tolerant systems. In: Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, HOTOS '99. IEEE Computer Society, Washington, DC, USA (1999)
26. Friendly, M., Denis, D.: The early origins and development of the scatterplot. Journal of the History of the Behavioral Sciences 41(2), 103-130 (2005)
27. Gelman, A., Hill, J.: Data Analysis using Regression and Multilevel/Hierarchical Models. Cambridge University Press (2006)
28. Golab, W., Li, X., Shah, M.A.: Analyzing consistency properties for fun and profit. In: Proceedings of the 30th Symposium on Principles of Distributed Computing (PODC), PODC '11, pp. 197-206. ACM, New York, NY, USA (2011). DOI 10.1145/1993806.1993834. URL http://doi.acm.org/10.1145/1993806.1993834
29. Gray, J.: The Benchmark Handbook for Database and Transaction Systems, 2nd edn., chap. Database and Transaction Processing Handbook. Morgan Kaufmann (1993)
30. Han, J., Kamber, M., Pei, J.: Data Mining, 3rd edn. Concepts and Techniques. Elsevier Inc. (2011)
31. Herlihy, M.P., Wing, J.M.: Linearizability: A correctness condition for concurrent objects. ACM Transactions on Programming Languages and Systems 12(3), 463-492 (1990). DOI 10.1145/78969.78972. URL http://doi.acm.org/10.1145/78969.78972
32. Hintze, J.L., Nelson, R.D.: Violin Plots: A Box Plot-Density Trace Synergism. The American Statistician 52(2), 181-184 (1998). DOI 10.1080/00031305.1998.10480559. URL http: //www.tandfonline.com/doi/abs/10.1080/00031305.1998.10480559
33. Huppler, K.: The art of building a good benchmark. In: Proceedings of the First TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2009), pp. 18-30. Springer (2009). DOI 10.1007/978-3-642-10424-4 3
34. IEEE: Standard 1061-1998, standard for a software quality metrics methodology (1998)
35. Izrailevsky, Y., Tseitlin, A.: The netflix simian army. http://techblog.netflix. com/2011/07/netflix-simian-army.html (accessed Jun 13, 2016) (2011)
36. Kampstra, P., et al.: Beanplot: A boxplot alternative for visual comparison of distributions. Journal of statistical software 28(1), 1-9 (2008)
37. Kaner, C., Bond, W.P.: Software engineering metrics: What do they measure and how do we know? In: Proceedings of the 10th International Software Metrics Symposium (METRICS) (2004)
38. Keller, A., Ludwig, H.: The wsla framework: Specifying and monitoring service level agreements for web services. Journal of Network and Systems Management 11(1), 57-81 (2003)
39. Kistowski, J.V., Arnold, J.A., Huppler, K., Lange, K.D., Henning, J.L., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), pp. 333-336. ACM (2015). DOI 10.1145/2668930.2688819
40. Klems, M., Bermbach, D., Weinert, R.: A runtime quality measurement framework for cloud database service systems. In: Proceedings of the 8th International Conference on the Quality of Information and Communications Technology (QUATIC), pp. 38-46 (2012). DOI 10.1109/ QUATIC.2012.17
41. Klems, M., Menzel, M., Fischer, R.: Consistency benchmarking: Evaluating the consistency behavior of middleware services in the cloud. In: P. Maglio, M. Weske, J. Yang, M. Fantinato (eds.) Service-Oriented Computing, Lecture Notes in Computer Science, vol. 6470, pp. 627-634. Springer Berlin Heidelberg (2010). DOI 10.1007/978-3-642-17358-5 48. URL http: //dx.doi.org/10.1007/978-3-642-17358-5_48
42. Kossmann, D., Kraska, T., Loesing, S.: An evaluation of alternative architectures for transaction processing in the cloud. In: Proceedings of the 30th International Conference on Management of Data (SIGMOD), SIGMOD '10, pp. 579-590. ACM, New York, NY, USA (2010). DOI 10.1145/1807167.1807231. URL http://doi.acm.org/10.1145/1807167. 1807231
43. Kuhlenkamp, J., Klems, M., Röss, O.: Benchmarking Scalability and Elasticity of Distributed Database Systems. In: Proceedings of the International Conference on Very Large Databases (VLDB), pp. 1219-1230 (2014)
44. Kuhlenkamp, J., Rudolph, K., Bermbach, D.: Service-Oriented Computing: 13th International Conference, ICSOC 2015, Goa, India, November 16-19, 2015, Proceedings, chap. AISLE: Assessment of Provisioned Service Levels in Public IaaS-Based Database Systems, pp. 154-168. Springer (2015). DOI 10.1007/978-3-662-48616-0 10
45. Kurze, T., Klems, M., Bermbach, D., Lenk, A., Tai, S., Kunze, M.: Cloud Federation. In: The Second International Conference on Cloud Computing, GRIDs, and Virtualization (CLOUD COMPUTING 2011). IARIA (2011)
46. Lakshman, A., Malik, P.: Cassandra: A decentralized structured storage system. SIGOPS Operating Systems Review 44(2), 35-40 (2010). DOI 10.1145/1773912.1773922. URL http://doi.acm.org/10.1145/1773912.1773922
47. Lamport, L.: The part-time parliament. ACM Transactions on Computer Systems 16(2), 133-169 (1998). DOI 10.1145/279227.279229. URL http://doi.acm.org/10.1145/ 279227.279229
48. Lenk, A., Menzel, M., Lipsky, J., Tai, S., Offermann, P.: What are you paying for? performance benchmarking for infrastructure-as-a-service offerings. In: Cloud Computing (CLOUD), 2011 IEEE International Conference on, pp. 484-491. IEEE (2011)
49. Linoff, G.S.: Data analysis using SQL and Excel. John Wiley & Sons (2015)
50. Ludwig, H., Keller, A., Dan, A., King, R.P., Franck, R.: Web service level agreement (wsla -language specification (2001)
51. Mell, P., Grance, T.: The NIST definition of cloud computing. NIST Special Publication 800-145 (2011)
52. Millard, C.: Identifying and managing legal risks in cloud computing. In: Proceedings of the 1st IEEE International Workshop on Legal and Technical Issues in Cloud Computing (CLaw 2015) (2015)
53. Müller, S., Bermbach, D., Tai, S., Pallas, F.: Benchmarking the performance impact of transport layer security in cloud database systems. In: Proceedings of the 2nd International Conference on Cloud Engineering (IC2E). IEEE (2014)
54. Nielsen, M.: Neural Networks and Deep Learning. Determination Press (2015). URL http: //neuralnetworksanddeeplearning.com/
55. Pallas, F., Bermbach, D., Müller, S., Tai, S.: Evidence-based security configurations for cloud datastores. In: Proceedings of the Symposium on Applied Computing, DADS track (SAC). ACM (2017)
56. Pallas, F., Günther, J., Bermbach, D.: Pick your choice in hbase: Security of performance. In: Proceedings of the International Conference on Big Data (IC2E). IEEE (2016)
57. Patel, P., Ranabahu, A.H., Sheth, A.P.: Service level agreement in cloud computing. Technical Report, Wright University (2009)
58. Patil, S., Polte, M., Ren, K., Tantisiriroj,W., Xiao, L., Ĺopez, J., Gibson, G., Fuchs, A., Rinaldi, B.: Ycsb++: Benchmarking and performance debugging advanced features in scalable table stores. In: Proceedings of the 2nd Symposium on Cloud Computing (SOCC), SOCC '11, pp. 9:1-9:14. ACM, New York, NY, USA (2011). DOI 10.1145/2038916.2038925. URL http://doi.acm.org/10.1145/2038916.2038925
59. Rabl, T., Frank, M., Sergieh, H.M., Kosch, H.: A data generator for cloud-scale benchmarking. In: Proceedings of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC), pp. 41-56. Springer (2010)
60. Rabl, T., Ǵomez-Villamor, S., Sadoghi, M., Muntes-Mulero, V., Jacobsen, H.A., Mankovskii, S.: Solving big data challenges for enterprise application performance management. Proceedings of the VLDB Endowment 5(12), 1724-1735 (2012). URL http://dl.acm.org/ citation.cfm?id=2367502.2367512
61. Raj Jain: Art of Computer Systems Performance Analysis Techniques For Experimental Design Measurements Simulation And Modeling (1991)
62. Ram, S., Liu, J.: A New Perspective on Semantics of Data Provenance. In: Proceedings of the First International Conference on Semantic Web in Provenance Management -Volume 526, SWPM'09, pp. 35-40. CEUR-WS.org, Aachen, Germany, Germany (2009). URL http: //dl.acm.org/citation.cfm?id=2889875.2889882
63. Salkind, N.J.: Encyclopedia of research design, vol. 1. Sage (2010)
64. Schroeder, B., Wierman, A., Harchol-Balter, M.: Open versus closed: A cautionary tale. In: Proceedings of the 3rd Conference on Networked Systems Design & Implementation, NSDI'06, pp. 18-18. USENIX Association (2006)
65. Strunk, A.: Qos-aware service composition: A survey. In: Web Services (ECOWS), 2010 IEEE 8th European Conference on, pp. 67-74 (2010). DOI 10.1109/ECOWS.2010.16
66. Tanenbaum, A.S., Steen, M.v.: Distributed Systems: Principles and Paradigms, 2nd edn. Pearson, Prentice Hall, Upper Saddle River, NJ (2007)
67. Terry, D., Demers, A., Petersen, K., Spreitzer, M., Theimer, M.,Welch, B.: Session guarantees for weakly consistent replicated data. In: Proceedings of the 3rd International Conference on Parallel and Distributed Information Systems (PDGC), pp. 140-149 (1994). DOI 10.1109/ PDIS.1994.331722
68. Tukey, J.W.: Exploratory Data Analysis (1977)
69. Vogels, W.: Eventually consistent. ACM Queue 6(6), 14-19 (2008). DOI 10.1145/1466443.1466448. URL http://doi.acm.org/10.1145/1466443.1466448
70. Waage, T.,Wiese, L.: Benchmarking encrypted data storage in hbase and cassandra with ycsb. In: International Symposium on Foundations and Practice of Security (FPS), pp. 311-325. Springer (2014)
71. Wada, H., Fekete, A., Zhao, L., Lee, K., Liu, A.: Data consistency properties and the tradeoffs in commercial cloud storages: the consumers' perspective. In: Proceedings of the 5th Conference on Innovative Data Systems Research (CIDR), pp. 134-143 (2011)
72. Wei, W.W.S.: Time Series Analysis: Univariate and Multivariate Methods, 2nd edn. Pearson Addison-Wesley (2006)
73. Welsh, M., Culler, D., Brewer, E.: Seda: An architecture for well-conditioned, scalable internet services. In: Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP), SOSP '01, pp. 230-243. ACM, New York, NY, USA (2001). DOI 10.1145/502034.502057. URL http://doi.acm.org/10.1145/502034.502057
74. Wingerath, W., Friedrich, S., Gessert, F., Ritter, N.: Who watches the watchmen? on the lack of validation in nosql benchmarking. In: Proceedings of Datenbanksysteme für Business, Technologie und Web (BTW 2015), 16. Fachtagung des GI-Fachbereichs Datenbanken und Informationssysteme (DBIS), pp. 351-360. GI Bonn (2015)
75. Wittern, E., Fischer, R.: "A Life-Cycle Model for Software Service Engineering pp. 164-171. Springer Berlin Heidelberg, Berlin, Heidelberg (2013). DOI 10.1007/ 978-3-642-40651-5 13. URL http://dx.doi.org/10.1007/978-3-642-40651-5_13
76. Wittern, E., Zirpins, C.: Service feature modeling: modeling and participatory ranking of service design alternatives. Software & Systems Modeling 15(2), 553-578 (2016)
77. Zellag, K., Kemme, B.: How consistent is your cloud application? In: Proceedings of the 3rd Symposium on Cloud Computing (SOCC), SOCC '12, pp. 6:1-6:14. ACM, New York, NY, USA (2012). DOI 10.1145/2391229.2391235. URL http://doi.acm.org/10.1145/ 2391229.2391235