Iron: Isolating network-based CPU in container environments

Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2018

Volumn , Issue , 2018, Pages 313-328

  Khalid, Junaid ^a   Rozner, Eric ^b   Felter, Wesley ^b   Xu, Cong ^b   Rajamani, Karthick ^b   Ferreira, Alexandre ^c   Akella, Aditya ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b IBM RESEARCH   (United States)

^c Arm Research

Author keywords

[No Author keywords available]

Indexed keywords

IRON; SYSTEMS ANALYSIS;

CLOUD PROVIDERS; COMPUTATIONAL OVERHEADS; ENFORCEMENT MECHANISMS; HEAVY NETWORK TRAFFIC; MULTI TENANTS; NETWORK STACK; PROCESSING NETWORK; RELIABLE PERFORMANCE;

CONTAINERS;

EID: 85076740772     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (76)

1. Docker swarm. https://github.com/docker/swarm. Accessed: 2017-09-25.
2. Linux advanced routing and traffic control howto. http://lartc.org/lartc.html. Accessed: 2017-09-25.
3. Linux control groups. https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt. Accessed: 2017-09-21.
4. Networking napi. https://wiki.linuxfoundation.org/networking/napi. Accessed: 2017-09-25.
5. perf: Linux profiling with performance counters. https://perf.wiki.kernel.org/index.php/Main_Page. Accessed: 2017-09-21.
6. ALIZADEH, M., EDSALL, T., DHARMAPURIKAR, S., VAIDYANATHAN, R., CHU, K., FINGERHUT, A., MATUS, F., PAN, R., YADAV, N., VARGHESE, G., ET AL. CONGA: Distributed Congestion-aware Load Balancing for Datacenters. In SIGCOMM (2014).
7. ALIZADEH, M., GREENBERG, A., MALTZ, D. A., PADHYE, J., PATEL, P., PRABHAKAR, B., SENGUPTA, S., AND SRIDHARAN, M. Data Center TCP (DCTCP). In SIGCOMM (2010).
8. AMAZON WEB SERVIES, INC. AWS Lambda: Serverless computing. https://aws.amazon.com/lambda/.
9. BALLANI, H., COSTA, P., KARAGIANNIS, T., AND ROWSTRON, A. Towards predictable datacenter networks. In ACM SIGCOMM Computer Communication Review (2011), vol. 41, ACM, pp. 242-253.
10. BANGA, G., DRUSCHEL, P., AND MOGUL, J. C. Resource containers: A new facility for resource management in server systems. In OSDI (1999), vol. 99, pp. 45-58.
11. BARTOLINI, D. B., SIRONI, F., SCIUTO, D., AND SANTAMBROGIO, M. D. Automated fine-grained cpu provisioning for virtual machines. ACM Trans. Archit. Code Optim. 11, 3 (July 2014), 27:1-27:25.
12. BELAY, A., PREKAS, G., KLIMOVIC, A., GROSSMAN, S., KOZYRAKIS, C., AND BUGNION, E. IX: A protected dataplane operating system for high throughput and low latency. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) (CO, 2014), USENIX Association, pp. 49-65.
13. BREWER, E. A. Kubernetes and the path to cloud native. In Proceedings of the Sixth ACM Symposium on Cloud Computing (New York, NY, USA, 2015), SoCC'15, ACM, pp. 167-167.
14. BRUTLAG, J. Speed Matters for Google Web Search. Tech. rep., 2009. https://services.google.com/fh/files/blogs/google_delayexp.pdf.
15. CHERKASOVA, L., GUPTA, D., AND VAHDAT, A. Comparison of the three cpu schedulers in xen. SIGMETRICS Perform. Eval. Rev. 35, 2 (Sept. 2007), 42-51.
16. CORBET, J. Jls2009: Generic receive offload. Linux Weekly News (LWN) (Oct. 2009). https://lwn.net/Articles/358910/.
17. CORBET, J. Software interrupts and realtime. Linux Weekly News (LWN) (Oct. 2012). https://lwn.net/Articles/520076/.
18. CORBET, J. Bulk network packet transmission. Linux Weekly News (LWN) (Oct. 2014). https://lwn.net/Articles/615238/.
19. CRONKITE-RATCLIFF, B., BERGMAN, A., VARGAFTIK, S., RAVI, M., MCKEOWN, N., ABRAHAM, I., AND KESLASSY, I. Virtualized congestion control. In Proceedings of the 2016 ACM SIGCOMM Conference (New York, NY, USA, 2016), SIGCOMM'16, ACM, pp. 230-243.
20. DEAN, J., AND BARROSO, L. A. The tail at scale. Communications of the ACM 56 (2013), 74-80.
21. DELIMITROU, C., AND KOZYRAKIS, C. Paragon: Qos-aware scheduling for heterogeneous datacenters. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (New York, NY, USA, 2013), ASPLOS'13, ACM, pp. 77-88.
22. DELIMITROU, C., AND KOZYRAKIS, C. Quasar: Resource-efficient and qos-aware cluster management. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (New York, NY, USA, 2014), ASPLOS'14, ACM, pp. 127-144.
23. DELIMITROU, C., AND KOZYRAKIS, C. Bolt: I know what you did last summer... in the cloud. SIGARCH Comput. Archit. News 45, 1 (Apr. 2017), 599-613.
24. DOBRESCU, M., ARGYRAKI, K., AND RATNASAMY, S. Toward predictable performance in software packet-processing platforms. In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12) (San Jose, CA, 2012), USENIX, pp. 141-154.
25. DRUSCHEL, P., AND BANGA, G. Lazy receiver processing (lrp): A network subsystem architecture for server systems. In OSDI (1996), vol. 96, pp. 261-275.
26. FONSECA, R., DUTTA, P., LEVIS, P., AND STOICA, I. Quanto: Tracking energy in networked embedded systems. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (Berkeley, CA, USA, 2008), OSDI'08, USENIX Association, pp. 323-338.
27. FUNARO, L., BEN-YEHUDA, O. A., AND SCHUSTER, A. Ginseng: Market-driven llc allocation. In 2016 USENIX Annual Technical Conference (USENIX ATC 16) (Denver, CO, 2016), USENIX Association, pp. 295-308.
28. GANGER, G. R., ENGLER, D. R., KAASHOEK, M. F., BRICEÑO, H. M., HUNT, R., AND PINCKNEY, T. Fast and flexible application-level networking on exokernel systems. ACM Trans. Comput. Syst. 20, 1 (Feb. 2002), 49-83.
29. GHODSI, A., SEKAR, V., ZAHARIA, M., AND STOICA, I. Multi-resource fair queueing for packet processing. In Proceedings of the ACM SIGCOMM 2012 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (New York, NY, USA, 2012), SIGCOMM'12, ACM, pp. 1-12.
30. GLEIXNER, T. [announce] 3.6.1-rt1. Linux Weekly News (LWN) (Oct. 2012). https://lwn.net/Articles/518993/.
31. GOOGLE INC. Cloud functions. https://cloud.google.com/functions/.
32. GULATI, A., MERCHANT, A., AND VARMAN, P. J. mclock: Handling throughput variability for hypervisor io scheduling. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation (Berkeley, CA, USA, 2010), OSDI'10, USENIX Association, pp. 437-450.
33. GUO, C., LU, G., WANG, H. J., YANG, S., KONG, C., SUN, P., WU, W., AND ZHANG, Y. Secondnet: a data center network virtualization architecture with bandwidth guarantees. In Proceedings of the 6th International Conference (2010), ACM, p. 15.
34. GUPTA, D., CHERKASOVA, L., GARDNER, R., AND VAHDAT, A. Enforcing performance isolation across virtual machines in xen. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing (2006), Springer, pp. 342-362.
35. HAN, S., MARSHALL, S., CHUN, B.-G., AND RATNASAMY, S. Megapipe: A new programming interface for scalable network i/o. In Presented as part of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12) (Hollywood, CA, 2012), USENIX, pp. 135-148.
36. HE, K., ROZNER, E., AGARWAL, K., FELTER, W., CARTER, J., AND AKELLA, A. Presto: Edge-based load balancing for fast datacenter networks. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (New York, NY, USA, 2015), SIGCOMM'15, ACM, pp. 465-478.
37. HE, K., ROZNER, E., AGARWAL, K., GU, Y. J., FELTER, W., CARTER, J., AND AKELLA, A. Ac/dc tcp: Virtual congestion control enforcement for datacenter networks. In Proceedings of the 2016 ACM SIGCOMM Conference (New York, NY, USA, 2016), SIGCOMM'16, ACM, pp. 244-257.
38. HENDRICKSON, S., STURDEVANT, S., HARTER, T., VENKATARAMANI, V., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Serverless computation with openlambda. In Proceedings of HotCloud (June 2016).
39. HERBERT, T., AND DE BRUIJN, W. Scaling in the linux networking stack, 2011. https://www.kernel.org/doc/ Documentation/networking/scaling.txt.
40. IYER, R., ZHAO, L., GUO, F., ILLIKKAL, R., MAKINENI, S., NEWELL, D., SOLIHIN, Y., HSU, L., AND REINHARDT, S. Qos policies and architecture for cache/memory in cmp platforms. In Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (New York, NY, USA, 2007), SIGMETRICS'07, ACM, pp. 25-36.
41. JEONG, E., WOOD, S., JAMSHED, M., JEONG, H., IHM, S., HAN, D., AND PARK, K. mtcp: a highly scalable user-level tcp stack for multicore systems. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14) (Seattle, WA, 2014), USENIX Association, pp. 489-502.
42. JEYAKUMAR, V., ALIZADEH, M., MAZIERES, D., PRABHAKAR, B., KIM, C., AND AZURE, W. Eyeq: Practical network performance isolation for the multi-tenant cloud. In HotCloud (2012).
43. JEYAKUMAR, V., ALIZADEH, M., MAZIÈRES, D., PRABHAKAR, B., KIM, C., AND GREENBERG, A. EyeQ: Practical Network Performance Isolation at the Edge. In NSDI (2013).
44. KASTURE, H., AND SANCHEZ, D. Ubik: Efficient cache sharing with strict qos for latency-critical workloads. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (New York, NY, USA, 2014), ASPLOS'14, ACM, pp. 729-742.
45. KAUFMANN, A., PETER, S., SHARMA, N. K., ANDERSON, T., AND KRISHNAMURTHY, A. High performance packet processing with flexnic. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems (New York, NY, USA, 2016), ASPLOS'16, ACM, pp. 67-81.
46. KOPYTOV, A. Sysbench manual. MySQL AB (2012).
47. KULKARNI, S. G., ZHANG, W., HWANG, J., RAJAGOPALAN, S., RAMAKRISHNAN, K. K., WOOD, T., ARUMAITHURAI, M., AND FU, X. Nfvnice: Dynamic backpressure and scheduling for nfv service chains. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2017), SIGCOMM'17, ACM, pp. 71-84.
48. LESLIE, I. M., MCAULEY, D., BLACK, R., ROSCOE, T., BARHAM, P., EVERS, D., FAIRBAIRNS, R., AND HYDEN, E. The design and implementation of an operating system to support distributed multimedia applications. IEEE J.Sel. A. Commun. 14, 7 (Sept. 2006), 1280-1297.
49. LO, D., CHENG, L., GOVINDARAJU, R., RANGANATHAN, P., AND KOZYRAKIS, C. Improving resource efficiency at scale with heracles. ACM Trans. Comput. Syst. 34, 2 (May 2016), 6:1-6:33.
50. LU, L., ZHANG, Y., DO, T., AL-KISWANY, S., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Physical Disentanglement in a Container-Based File System. In Proceedings of the 11th Symposium on Operating Systems Design and Implementation (OSDI'14) (Broomfield, CO, October 2014).
51. MACE, J., BODIK, P., FONSECA, R., AND MUSUVATHI, M. Retro: Targeted resource management in multi-tenant distributed systems. In NSDI (2015), pp. 589-603.
52. MARTINS, J., AHMED, M., RAICIU, C., OLTEANU, V., HONDA, M., BIFULCO, R., AND HUICI, F. Clickos and the art of network function virtualization. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14) (Seattle, WA, 2014), USENIX Association, pp. 459-473.
53. MATTHEWS, J. N., HU, W., HAPUARACHCHI, M., DESHANE, T., DIMATOS, D., HAMILTON, G., MCCABE, M., AND OWENS, J. Quantifying the performance isolation properties of virtualization systems. In Proceedings of the 2007 Workshop on Experimental Computer Science (New York, NY, USA, 2007), ExpCS '07, ACM.
54. MCCULLOUGH, J. C., DUNAGAN, J., WOLMAN, A., AND SNOEREN, A. C. Stout: An adaptive interface to scalable cloud storage. In Proc. of the USENIX Annual Technical Conference-ATC (2010), pp. 47-60.
55. MICROSOFT CORP. Azure functions. https://azure.microsoft.com/en-us/services/functions/.
56. MUNDADA, Y., RAMACHANDRAN, A., AND FEAMSTER, N. Silverline: Data and network isolation for cloud services. In HotCloud (2011).
57. PETER, S., LI, J., ZHANG, I., PORTS, D. R. K., WOOS, D., KRISHNAMURTHY, A., ANDERSON, T., AND ROSCOE, T. Arrakis: The operating system is the control plane. ACM Trans. Comput. Syst. 33, 4 (Nov. 2015), 11:1-11:30.
58. POPA, L., KUMAR, G., CHOWDHURY, M., KRISHNAMURTHY, A., RATNASAMY, S., AND STOICA, I. FairCloud: Sharing the Network in Cloud Computing. In SIGCOMM (2012).
59. POPA, L., YALAGANDULA, P., BANERJEE, S., MOGUL, J. C., TURNER, Y., AND SANTOS, J. R. Elasticswitch: Practical work-conserving bandwidth guarantees for cloud computing. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 351-362.
60. PORTER, D. E., BOYD-WICKIZER, S., HOWELL, J., OLINSKY, R., AND HUNT, G. Rethinking the library os from the top down. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (March 2011), ACM.
61. RADHAKRISHNAN, S., GENG, Y., JEYAKUMAR, V., KABBANI, A., PORTER, G., AND VAHDAT, A. SENIC: Scalable NIC for End-host Rate Limiting. In NSDI (2014).
62. REISS, C., TUMANOV, A., GANGER, G. R., KATZ, R. H., AND KOZUCH, M. A. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of the Third ACM Symposium on Cloud Computing (New York, NY, USA, 2012), SoCC'12, ACM, pp. 7:1-7:13.
63. RODRIGUES, H., SANTOS, J. R., TURNER, Y., SOARES, P., AND GUEDES, D. O. Gatekeeper: Supporting bandwidth guarantees for multi-tenant datacenter networks. In WIOV (2011).
64. SHARMA, P., CHAUFOURNIER, L., SHENOY, P., AND TAY, Y. C. Containers and virtual machines at scale: A comparative study. In Proceedings of the 17th International Middleware Conference (New York, NY, USA, 2016), Middleware'16, ACM, pp. 1:1-1:13.
65. SHIEH, A., KANDULA, S., GREENBERG, A. G., AND KIM, C. Seawall: Performance isolation for cloud datacenter networks. In HotCloud (2010).
66. THERESKA, E., BALLANI, H., O'SHEA, G., KARAGIANNIS, T., ROWSTRON, A., TALPEY, T., BLACK, R., AND ZHU, T. Ioflow: A software-defined storage architecture. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (New York, NY, USA, 2013), SOSP'13, ACM, pp. 182-196.
67. TURNER, P., RAO, B. B., AND RAO, N. Cpu bandwidth control for cfs. In Proceedings of the Linux Symposium (2010), pp. 245-254.
68. VASILESCU, L., OLTEANU, V., AND RAICIU, C. Sharing cpus via endpoint congestion control. In Proceedings of the Workshop on Kernel-Bypass Networks (New York, NY, USA, 2017), KBNets'17, ACM, pp. 31-36.
69. VERMA, A., PEDROSA, L., KORUPOLU, M. R., OPPENHEIMER, D., TUNE, E., AND WILKES, J. Large-scale cluster management at Google with Borg. In Proceedings of the European Conference on Computer Systems (EuroSys) (Bordeaux, France, 2015).
70. WACHS, M., ABD-EL-MALEK, M., THERESKA, E., AND GANGER, G. R. Argon: Performance insulation for shared storage servers. In Proceedings of the 5th USENIX Conference on File and Storage Technologies (Berkeley, CA, USA, 2007), FAST'07, USENIX Association, pp. 5-5.
71. YANG, H., BRESLOW, A., MARS, J., AND TANG, L. Bubble-flux: Precise online qos management for increased utilization in warehouse scale computers. In Proceedings of the 40th Annual International Symposium on Computer Architecture (New York, NY, USA, 2013), ISCA'13, ACM, pp. 607-618.
72. YANG, S., HARTER, T., AGRAWAL, N., KOWSALYA, S. S., KRISHNAMURTHY, A., AL-KISWANY, S., KAUSHIK, R. T., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Split-level i/o scheduling. In Proceedings of the 25th Symposium on Operating Systems Principles (2015), ACM, pp. 474-489.
73. YOSIFOVICH, P., IONESCU, A., RUSSINOVICH, M. E., AND SOLOMON, D. A. Windows Internals, Part 1: System architecture, processes, threads, memory management, and more, 7th ed. Microsoft Press, 2017.
74. ZHANG, W., RAJASEKARAN, S., DUAN, S., WOOD, T., AND ZHUY, M. Minimizing interference and maximizing progress for hadoop virtual machines. SIGMETRICS Perform. Eval. Rev. 42, 4 (June 2015), 62-71.
75. ZHANG, Y., AND WEST, R. Process-aware interrupt scheduling and accounting. In Proceedings of the 27th IEEE International Real-Time Systems Symposium (Washington, DC, USA, 2006), RTSS'06, IEEE Computer Society, pp. 191-201.
76. ZILBERMAN, N., AUDZEVICH, Y., COVINGTON, G. A., AND MOORE, A. W. Netfpga sume: Toward 100 gbps as research commodity. IEEE Micro 34, 5 (2014), 32-41.