論文閱讀: Taming Memory Related Performance Pitfalls in Linux Cgroups

Title: Taming Memory Related Performance Pitfalls in Linux Cgroups
Zhenyun Zhuang, Cuong Tran, Jerry Weng, Haricharan Ramachandra, Badri Sridharan 2029 Stierlin Ct, Mountain View, CA 94043, USA {zzhuang, ctran, jweng, hramachandra, bsridharan}@linkedin.com

鑒於近期新的軟體接建構於Linux CGroup的功能上(例如Docker)來分配CPU以及Memory資源，但在Memory的資源分配上筆者提出下列問題

1. Memory is not reserved for cgroups:
   Cgroup只能設定最高上限但沒辦法Reserve memory。
2. Both anonymous memory and page cache are part of memory limit and the former can evict the latter:
   Cgroup在計算Memory Limitation時，將Anonymous(Stack/Heap) + Page Cache一起計算，會造成排擠效應。
3. OS can steal page cache from any cgroups:
   Cgroup沒辦法控制哪些Memory會被Reclaim。
4. OS can swap any cgroups:
   Cgroup沒辦法控制swap policy。

Cgroups illustration

1. Root cgroup is unbounded: No memory limitation.
2. Root cgroup and Regular cgroup could have only one swappiness.
   Thus, unlimit root cgroup could make the other regular cgroup makes low performance.

Strategy for pitfalls

1. (Regular cgroups) Pre-touching the requested memory:
   使用Java: AlwaysPreTouchFlag JVM啟動時就分配所有Java Heap， AlwaysPreTouchSince the memory limit of cgroups is not allocated beforehand, it helps to “pre-touch” the requested anonymous memory and avoid use-as-you-go requests.
2. (Application onboarding) Estimating page cache to decide memory limit:
   Since a cgroup’s memory limit includes both anonymous memory and page cache used by the processes in the cgroup, deploying applications to cgroups should consider an application’s memory usage of both types. Unfortunately estimating the page cache usage for an application is very difficult, as there is no direct Linux metric on page cache usage by processes. Moreover, there are other mechanisms that further complicate the issue.
3. (Root cgroup) Isolating memory usage of system utilities and housekeeping processes:
   限制system process usage like sshd, cround
   One of the causes of the performance issues is the unbounded memory usage of root cgroup. Different from typical applications, system or housekeeping processes (sshd, crond) by default run in root cgroup