698 lines · plain
1==============2Control Groups3==============4 5Written by Paul Menage <menage@google.com> based on6Documentation/admin-guide/cgroup-v1/cpusets.rst7 8Original copyright statements from cpusets.txt:9 10Portions Copyright (C) 2004 BULL SA.11 12Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.13 14Modified by Paul Jackson <pj@sgi.com>15 16Modified by Christoph Lameter <cl@linux.com>17 18.. CONTENTS:19 20 1. Control Groups21 1.1 What are cgroups ?22 1.2 Why are cgroups needed ?23 1.3 How are cgroups implemented ?24 1.4 What does notify_on_release do ?25 1.5 What does clone_children do ?26 1.6 How do I use cgroups ?27 2. Usage Examples and Syntax28 2.1 Basic Usage29 2.2 Attaching processes30 2.3 Mounting hierarchies by name31 3. Kernel API32 3.1 Overview33 3.2 Synchronization34 3.3 Subsystem API35 4. Extended attributes usage36 5. Questions37 381. Control Groups39=================40 411.1 What are cgroups ?42----------------------43 44Control Groups provide a mechanism for aggregating/partitioning sets of45tasks, and all their future children, into hierarchical groups with46specialized behaviour.47 48Definitions:49 50A *cgroup* associates a set of tasks with a set of parameters for one51or more subsystems.52 53A *subsystem* is a module that makes use of the task grouping54facilities provided by cgroups to treat groups of tasks in55particular ways. A subsystem is typically a "resource controller" that56schedules a resource or applies per-cgroup limits, but it may be57anything that wants to act on a group of processes, e.g. a58virtualization subsystem.59 60A *hierarchy* is a set of cgroups arranged in a tree, such that61every task in the system is in exactly one of the cgroups in the62hierarchy, and a set of subsystems; each subsystem has system-specific63state attached to each cgroup in the hierarchy. Each hierarchy has64an instance of the cgroup virtual filesystem associated with it.65 66At any one time there may be multiple active hierarchies of task67cgroups. Each hierarchy is a partition of all tasks in the system.68 69User-level code may create and destroy cgroups by name in an70instance of the cgroup virtual file system, specify and query to71which cgroup a task is assigned, and list the task PIDs assigned to72a cgroup. Those creations and assignments only affect the hierarchy73associated with that instance of the cgroup file system.74 75On their own, the only use for cgroups is for simple job76tracking. The intention is that other subsystems hook into the generic77cgroup support to provide new attributes for cgroups, such as78accounting/limiting the resources which processes in a cgroup can79access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow80you to associate a set of CPUs and a set of memory nodes with the81tasks in each cgroup.82 83.. _cgroups-why-needed:84 851.2 Why are cgroups needed ?86----------------------------87 88There are multiple efforts to provide process aggregations in the89Linux kernel, mainly for resource-tracking purposes. Such efforts90include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server91namespaces. These all require the basic notion of a92grouping/partitioning of processes, with newly forked processes ending93up in the same group (cgroup) as their parent process.94 95The kernel cgroup patch provides the minimum essential kernel96mechanisms required to efficiently implement such groups. It has97minimal impact on the system fast paths, and provides hooks for98specific subsystems such as cpusets to provide additional behaviour as99desired.100 101Multiple hierarchy support is provided to allow for situations where102the division of tasks into cgroups is distinctly different for103different subsystems - having parallel hierarchies allows each104hierarchy to be a natural division of tasks, without having to handle105complex combinations of tasks that would be present if several106unrelated subsystems needed to be forced into the same tree of107cgroups.108 109At one extreme, each resource controller or subsystem could be in a110separate hierarchy; at the other extreme, all subsystems111would be attached to the same hierarchy.112 113As an example of a scenario (originally proposed by vatsa@in.ibm.com)114that can benefit from multiple hierarchies, consider a large115university server with various users - students, professors, system116tasks etc. The resource planning for this server could be along the117following lines::118 119 CPU : "Top cpuset"120 / \121 CPUSet1 CPUSet2122 | |123 (Professors) (Students)124 125 In addition (system tasks) are attached to topcpuset (so126 that they can run anywhere) with a limit of 20%127 128 Memory : Professors (50%), Students (30%), system (20%)129 130 Disk : Professors (50%), Students (30%), system (20%)131 132 Network : WWW browsing (20%), Network File System (60%), others (20%)133 / \134 Professors (15%) students (5%)135 136Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes137into the NFS network class.138 139At the same time Firefox/Lynx will share an appropriate CPU/Memory class140depending on who launched it (prof/student).141 142With the ability to classify tasks differently for different resources143(by putting those resource subsystems in different hierarchies),144the admin can easily set up a script which receives exec notifications145and depending on who is launching the browser he can::146 147 # echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks148 149With only a single hierarchy, he now would potentially have to create150a separate cgroup for every browser launched and associate it with151appropriate network and other resource class. This may lead to152proliferation of such cgroups.153 154Also let's say that the administrator would like to give enhanced network155access temporarily to a student's browser (since it is night and the user156wants to do online gaming :)) OR give one of the student's simulation157apps enhanced CPU power.158 159With ability to write PIDs directly to resource classes, it's just a160matter of::161 162 # echo pid > /sys/fs/cgroup/network/<new_class>/tasks163 (after some time)164 # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks165 166Without this ability, the administrator would have to split the cgroup into167multiple separate ones and then associate the new cgroups with the168new resource classes.169 170 171 1721.3 How are cgroups implemented ?173---------------------------------174 175Control Groups extends the kernel as follows:176 177 - Each task in the system has a reference-counted pointer to a178 css_set.179 180 - A css_set contains a set of reference-counted pointers to181 cgroup_subsys_state objects, one for each cgroup subsystem182 registered in the system. There is no direct link from a task to183 the cgroup of which it's a member in each hierarchy, but this184 can be determined by following pointers through the185 cgroup_subsys_state objects. This is because accessing the186 subsystem state is something that's expected to happen frequently187 and in performance-critical code, whereas operations that require a188 task's actual cgroup assignments (in particular, moving between189 cgroups) are less common. A linked list runs through the cg_list190 field of each task_struct using the css_set, anchored at191 css_set->tasks.192 193 - A cgroup hierarchy filesystem can be mounted for browsing and194 manipulation from user space.195 196 - You can list all the tasks (by PID) attached to any cgroup.197 198The implementation of cgroups requires a few, simple hooks199into the rest of the kernel, none in performance-critical paths:200 201 - in init/main.c, to initialize the root cgroups and initial202 css_set at system boot.203 204 - in fork and exit, to attach and detach a task from its css_set.205 206In addition, a new file system of type "cgroup" may be mounted, to207enable browsing and modifying the cgroups presently known to the208kernel. When mounting a cgroup hierarchy, you may specify a209comma-separated list of subsystems to mount as the filesystem mount210options. By default, mounting the cgroup filesystem attempts to211mount a hierarchy containing all registered subsystems.212 213If an active hierarchy with exactly the same set of subsystems already214exists, it will be reused for the new mount. If no existing hierarchy215matches, and any of the requested subsystems are in use in an existing216hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy217is activated, associated with the requested subsystems.218 219It's not currently possible to bind a new subsystem to an active220cgroup hierarchy, or to unbind a subsystem from an active cgroup221hierarchy. This may be possible in future, but is fraught with nasty222error-recovery issues.223 224When a cgroup filesystem is unmounted, if there are any225child cgroups created below the top-level cgroup, that hierarchy226will remain active even though unmounted; if there are no227child cgroups then the hierarchy will be deactivated.228 229No new system calls are added for cgroups - all support for230querying and modifying cgroups is via this cgroup file system.231 232Each task under /proc has an added file named 'cgroup' displaying,233for each active hierarchy, the subsystem names and the cgroup name234as the path relative to the root of the cgroup file system.235 236Each cgroup is represented by a directory in the cgroup file system237containing the following files describing that cgroup:238 239 - tasks: list of tasks (by PID) attached to that cgroup. This list240 is not guaranteed to be sorted. Writing a thread ID into this file241 moves the thread into this cgroup.242 - cgroup.procs: list of thread group IDs in the cgroup. This list is243 not guaranteed to be sorted or free of duplicate TGIDs, and userspace244 should sort/uniquify the list if this property is required.245 Writing a thread group ID into this file moves all threads in that246 group into this cgroup.247 - notify_on_release flag: run the release agent on exit?248 - release_agent: the path to use for release notifications (this file249 exists in the top cgroup only)250 251Other subsystems such as cpusets may add additional files in each252cgroup dir.253 254New cgroups are created using the mkdir system call or shell255command. The properties of a cgroup, such as its flags, are256modified by writing to the appropriate file in that cgroups257directory, as listed above.258 259The named hierarchical structure of nested cgroups allows partitioning260a large system into nested, dynamically changeable, "soft-partitions".261 262The attachment of each task, automatically inherited at fork by any263children of that task, to a cgroup allows organizing the work load264on a system into related sets of tasks. A task may be re-attached to265any other cgroup, if allowed by the permissions on the necessary266cgroup file system directories.267 268When a task is moved from one cgroup to another, it gets a new269css_set pointer - if there's an already existing css_set with the270desired collection of cgroups then that group is reused, otherwise a new271css_set is allocated. The appropriate existing css_set is located by272looking into a hash table.273 274To allow access from a cgroup to the css_sets (and hence tasks)275that comprise it, a set of cg_cgroup_link objects form a lattice;276each cg_cgroup_link is linked into a list of cg_cgroup_links for277a single cgroup on its cgrp_link_list field, and a list of278cg_cgroup_links for a single css_set on its cg_link_list.279 280Thus the set of tasks in a cgroup can be listed by iterating over281each css_set that references the cgroup, and sub-iterating over282each css_set's task set.283 284The use of a Linux virtual file system (vfs) to represent the285cgroup hierarchy provides for a familiar permission and name space286for cgroups, with a minimum of additional kernel code.287 2881.4 What does notify_on_release do ?289------------------------------------290 291If the notify_on_release flag is enabled (1) in a cgroup, then292whenever the last task in the cgroup leaves (exits or attaches to293some other cgroup) and the last child cgroup of that cgroup294is removed, then the kernel runs the command specified by the contents295of the "release_agent" file in that hierarchy's root directory,296supplying the pathname (relative to the mount point of the cgroup297file system) of the abandoned cgroup. This enables automatic298removal of abandoned cgroups. The default value of299notify_on_release in the root cgroup at system boot is disabled300(0). The default value of other cgroups at creation is the current301value of their parents' notify_on_release settings. The default value of302a cgroup hierarchy's release_agent path is empty.303 3041.5 What does clone_children do ?305---------------------------------306 307This flag only affects the cpuset controller. If the clone_children308flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its309configuration from the parent during initialization.310 3111.6 How do I use cgroups ?312--------------------------313 314To start a new job that is to be contained within a cgroup, using315the "cpuset" cgroup subsystem, the steps are something like::316 317 1) mount -t tmpfs cgroup_root /sys/fs/cgroup318 2) mkdir /sys/fs/cgroup/cpuset319 3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset320 4) Create the new cgroup by doing mkdir's and write's (or echo's) in321 the /sys/fs/cgroup/cpuset virtual file system.322 5) Start a task that will be the "founding father" of the new job.323 6) Attach that task to the new cgroup by writing its PID to the324 /sys/fs/cgroup/cpuset tasks file for that cgroup.325 7) fork, exec or clone the job tasks from this founding father task.326 327For example, the following sequence of commands will setup a cgroup328named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,329and then start a subshell 'sh' in that cgroup::330 331 mount -t tmpfs cgroup_root /sys/fs/cgroup332 mkdir /sys/fs/cgroup/cpuset333 mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset334 cd /sys/fs/cgroup/cpuset335 mkdir Charlie336 cd Charlie337 /bin/echo 2-3 > cpuset.cpus338 /bin/echo 1 > cpuset.mems339 /bin/echo $$ > tasks340 sh341 # The subshell 'sh' is now running in cgroup Charlie342 # The next line should display '/Charlie'343 cat /proc/self/cgroup344 3452. Usage Examples and Syntax346============================347 3482.1 Basic Usage349---------------350 351Creating, modifying, using cgroups can be done through the cgroup352virtual filesystem.353 354To mount a cgroup hierarchy with all available subsystems, type::355 356 # mount -t cgroup xxx /sys/fs/cgroup357 358The "xxx" is not interpreted by the cgroup code, but will appear in359/proc/mounts so may be any useful identifying string that you like.360 361Note: Some subsystems do not work without some user input first. For instance,362if cpusets are enabled the user will have to populate the cpus and mems files363for each new cgroup created before that group can be used.364 365As explained in section `1.2 Why are cgroups needed?` you should create366different hierarchies of cgroups for each single resource or group of367resources you want to control. Therefore, you should mount a tmpfs on368/sys/fs/cgroup and create directories for each cgroup resource or resource369group::370 371 # mount -t tmpfs cgroup_root /sys/fs/cgroup372 # mkdir /sys/fs/cgroup/rg1373 374To mount a cgroup hierarchy with just the cpuset and memory375subsystems, type::376 377 # mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1378 379While remounting cgroups is currently supported, it is not recommend380to use it. Remounting allows changing bound subsystems and381release_agent. Rebinding is hardly useful as it only works when the382hierarchy is empty and release_agent itself should be replaced with383conventional fsnotify. The support for remounting will be removed in384the future.385 386To Specify a hierarchy's release_agent::387 388 # mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \389 xxx /sys/fs/cgroup/rg1390 391Note that specifying 'release_agent' more than once will return failure.392 393Note that changing the set of subsystems is currently only supported394when the hierarchy consists of a single (root) cgroup. Supporting395the ability to arbitrarily bind/unbind subsystems from an existing396cgroup hierarchy is intended to be implemented in the future.397 398Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the399tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1400is the cgroup that holds the whole system.401 402If you want to change the value of release_agent::403 404 # echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent405 406It can also be changed via remount.407 408If you want to create a new cgroup under /sys/fs/cgroup/rg1::409 410 # cd /sys/fs/cgroup/rg1411 # mkdir my_cgroup412 413Now you want to do something with this cgroup:414 415 # cd my_cgroup416 417In this directory you can find several files::418 419 # ls420 cgroup.procs notify_on_release tasks421 (plus whatever files added by the attached subsystems)422 423Now attach your shell to this cgroup::424 425 # /bin/echo $$ > tasks426 427You can also create cgroups inside your cgroup by using mkdir in this428directory::429 430 # mkdir my_sub_cs431 432To remove a cgroup, just use rmdir::433 434 # rmdir my_sub_cs435 436This will fail if the cgroup is in use (has cgroups inside, or437has processes attached, or is held alive by other subsystem-specific438reference).439 4402.2 Attaching processes441-----------------------442 443::444 445 # /bin/echo PID > tasks446 447Note that it is PID, not PIDs. You can only attach ONE task at a time.448If you have several tasks to attach, you have to do it one after another::449 450 # /bin/echo PID1 > tasks451 # /bin/echo PID2 > tasks452 ...453 # /bin/echo PIDn > tasks454 455You can attach the current shell task by echoing 0::456 457 # echo 0 > tasks458 459You can use the cgroup.procs file instead of the tasks file to move all460threads in a threadgroup at once. Echoing the PID of any task in a461threadgroup to cgroup.procs causes all tasks in that threadgroup to be462attached to the cgroup. Writing 0 to cgroup.procs moves all tasks463in the writing task's threadgroup.464 465Note: Since every task is always a member of exactly one cgroup in each466mounted hierarchy, to remove a task from its current cgroup you must467move it into a new cgroup (possibly the root cgroup) by writing to the468new cgroup's tasks file.469 470Note: Due to some restrictions enforced by some cgroup subsystems, moving471a process to another cgroup can fail.472 4732.3 Mounting hierarchies by name474--------------------------------475 476Passing the name=<x> option when mounting a cgroups hierarchy477associates the given name with the hierarchy. This can be used when478mounting a pre-existing hierarchy, in order to refer to it by name479rather than by its set of active subsystems. Each hierarchy is either480nameless, or has a unique name.481 482The name should match [\w.-]+483 484When passing a name=<x> option for a new hierarchy, you need to485specify subsystems manually; the legacy behaviour of mounting all486subsystems when none are explicitly specified is not supported when487you give a subsystem a name.488 489The name of the subsystem appears as part of the hierarchy description490in /proc/mounts and /proc/<pid>/cgroups.491 492 4933. Kernel API494=============495 4963.1 Overview497------------498 499Each kernel subsystem that wants to hook into the generic cgroup500system needs to create a cgroup_subsys object. This contains501various methods, which are callbacks from the cgroup system, along502with a subsystem ID which will be assigned by the cgroup system.503 504Other fields in the cgroup_subsys object include:505 506- subsys_id: a unique array index for the subsystem, indicating which507 entry in cgroup->subsys[] this subsystem should be managing.508 509- name: should be initialized to a unique subsystem name. Should be510 no longer than MAX_CGROUP_TYPE_NAMELEN.511 512- early_init: indicate if the subsystem needs early initialization513 at system boot.514 515Each cgroup object created by the system has an array of pointers,516indexed by subsystem ID; this pointer is entirely managed by the517subsystem; the generic cgroup code will never touch this pointer.518 5193.2 Synchronization520-------------------521 522There is a global mutex, cgroup_mutex, used by the cgroup523system. This should be taken by anything that wants to modify a524cgroup. It may also be taken to prevent cgroups from being525modified, but more specific locks may be more appropriate in that526situation.527 528See kernel/cgroup.c for more details.529 530Subsystems can take/release the cgroup_mutex via the functions531cgroup_lock()/cgroup_unlock().532 533Accessing a task's cgroup pointer may be done in the following ways:534- while holding cgroup_mutex535- while holding the task's alloc_lock (via task_lock())536- inside an rcu_read_lock() section via rcu_dereference()537 5383.3 Subsystem API539-----------------540 541Each subsystem should:542 543- add an entry in linux/cgroup_subsys.h544- define a cgroup_subsys object called <name>_cgrp_subsys545 546Each subsystem may export the following methods. The only mandatory547methods are css_alloc/free. Any others that are null are presumed to548be successful no-ops.549 550``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)``551(cgroup_mutex held by caller)552 553Called to allocate a subsystem state object for a cgroup. The554subsystem should allocate its subsystem state object for the passed555cgroup, returning a pointer to the new object on success or a556ERR_PTR() value. On success, the subsystem pointer should point to557a structure of type cgroup_subsys_state (typically embedded in a558larger subsystem-specific object), which will be initialized by the559cgroup system. Note that this will be called at initialization to560create the root subsystem state for this subsystem; this case can be561identified by the passed cgroup object having a NULL parent (since562it's the root of the hierarchy) and may be an appropriate place for563initialization code.564 565``int css_online(struct cgroup *cgrp)``566(cgroup_mutex held by caller)567 568Called after @cgrp successfully completed all allocations and made569visible to cgroup_for_each_child/descendant_*() iterators. The570subsystem may choose to fail creation by returning -errno. This571callback can be used to implement reliable state sharing and572propagation along the hierarchy. See the comment on573cgroup_for_each_live_descendant_pre() for details.574 575``void css_offline(struct cgroup *cgrp);``576(cgroup_mutex held by caller)577 578This is the counterpart of css_online() and called iff css_online()579has succeeded on @cgrp. This signifies the beginning of the end of580@cgrp. @cgrp is being removed and the subsystem should start dropping581all references it's holding on @cgrp. When all references are dropped,582cgroup removal will proceed to the next step - css_free(). After this583callback, @cgrp should be considered dead to the subsystem.584 585``void css_free(struct cgroup *cgrp)``586(cgroup_mutex held by caller)587 588The cgroup system is about to free @cgrp; the subsystem should free589its subsystem state object. By the time this method is called, @cgrp590is completely unused; @cgrp->parent is still valid. (Note - can also591be called for a newly-created cgroup if an error occurs after this592subsystem's create() method has been called for the new cgroup).593 594``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``595(cgroup_mutex held by caller)596 597Called prior to moving one or more tasks into a cgroup; if the598subsystem returns an error, this will abort the attach operation.599@tset contains the tasks to be attached and is guaranteed to have at600least one task in it.601 602If there are multiple tasks in the taskset, then:603 - it's guaranteed that all are from the same thread group604 - @tset contains all tasks from the thread group whether or not605 they're switching cgroups606 - the first task is the leader607 608Each @tset entry also contains the task's old cgroup and tasks which609aren't switching cgroup can be skipped easily using the610cgroup_taskset_for_each() iterator. Note that this isn't called on a611fork. If this method returns 0 (success) then this should remain valid612while the caller holds cgroup_mutex and it is ensured that either613attach() or cancel_attach() will be called in future.614 615``void css_reset(struct cgroup_subsys_state *css)``616(cgroup_mutex held by caller)617 618An optional operation which should restore @css's configuration to the619initial state. This is currently only used on the unified hierarchy620when a subsystem is disabled on a cgroup through621"cgroup.subtree_control" but should remain enabled because other622subsystems depend on it. cgroup core makes such a css invisible by623removing the associated interface files and invokes this callback so624that the hidden subsystem can return to the initial neutral state.625This prevents unexpected resource control from a hidden css and626ensures that the configuration is in the initial state when it is made627visible again later.628 629``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``630(cgroup_mutex held by caller)631 632Called when a task attach operation has failed after can_attach() has succeeded.633A subsystem whose can_attach() has some side-effects should provide this634function, so that the subsystem can implement a rollback. If not, not necessary.635This will be called only about subsystems whose can_attach() operation have636succeeded. The parameters are identical to can_attach().637 638``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``639(cgroup_mutex held by caller)640 641Called after the task has been attached to the cgroup, to allow any642post-attachment activity that requires memory allocations or blocking.643The parameters are identical to can_attach().644 645``void fork(struct task_struct *task)``646 647Called when a task is forked into a cgroup.648 649``void exit(struct task_struct *task)``650 651Called during task exit.652 653``void free(struct task_struct *task)``654 655Called when the task_struct is freed.656 657``void bind(struct cgroup *root)``658(cgroup_mutex held by caller)659 660Called when a cgroup subsystem is rebound to a different hierarchy661and root cgroup. Currently this will only involve movement between662the default hierarchy (which never has sub-cgroups) and a hierarchy663that is being created/destroyed (and hence has no sub-cgroups).664 6654. Extended attribute usage666===========================667 668cgroup filesystem supports certain types of extended attributes in its669directories and files. The current supported types are:670 671 - Trusted (XATTR_TRUSTED)672 - Security (XATTR_SECURITY)673 674Both require CAP_SYS_ADMIN capability to set.675 676Like in tmpfs, the extended attributes in cgroup filesystem are stored677using kernel memory and it's advised to keep the usage at minimum. This678is the reason why user defined extended attributes are not supported, since679any user can do it and there's no limit in the value size.680 681The current known users for this feature are SELinux to limit cgroup usage682in containers and systemd for assorted meta data like main PID in a cgroup683(systemd creates a cgroup per service).684 6855. Questions686============687 688::689 690 Q: what's up with this '/bin/echo' ?691 A: bash's builtin 'echo' command does not check calls to write() against692 errors. If you use it in the cgroup file system, you won't be693 able to tell whether a command succeeded or failed.694 695 Q: When I attach processes, only the first of the line gets really attached !696 A: We can only return one error code per call to write(). So you should also697 put only ONE PID.698