575 lines · plain
1==============================2RT-mutex implementation design3==============================4 5Copyright (c) 2006 Steven Rostedt6 7Licensed under the GNU Free Documentation License, Version 1.28 9 10This document tries to describe the design of the rtmutex.c implementation.11It doesn't describe the reasons why rtmutex.c exists. For that please see12Documentation/locking/rt-mutex.rst. Although this document does explain problems13that happen without this code, but that is in the concept to understand14what the code actually is doing.15 16The goal of this document is to help others understand the priority17inheritance (PI) algorithm that is used, as well as reasons for the18decisions that were made to implement PI in the manner that was done.19 20 21Unbounded Priority Inversion22----------------------------23 24Priority inversion is when a lower priority process executes while a higher25priority process wants to run. This happens for several reasons, and26most of the time it can't be helped. Anytime a high priority process wants27to use a resource that a lower priority process has (a mutex for example),28the high priority process must wait until the lower priority process is done29with the resource. This is a priority inversion. What we want to prevent30is something called unbounded priority inversion. That is when the high31priority process is prevented from running by a lower priority process for32an undetermined amount of time.33 34The classic example of unbounded priority inversion is where you have three35processes, let's call them processes A, B, and C, where A is the highest36priority process, C is the lowest, and B is in between. A tries to grab a lock37that C owns and must wait and lets C run to release the lock. But in the38meantime, B executes, and since B is of a higher priority than C, it preempts C,39but by doing so, it is in fact preempting A which is a higher priority process.40Now there's no way of knowing how long A will be sleeping waiting for C41to release the lock, because for all we know, B is a CPU hog and will42never give C a chance to release the lock. This is called unbounded priority43inversion.44 45Here's a little ASCII art to show the problem::46 47 grab lock L1 (owned by C)48 |49 A ---+50 C preempted by B51 |52 C +----+53 54 B +-------->55 B now keeps A from running.56 57 58Priority Inheritance (PI)59-------------------------60 61There are several ways to solve this issue, but other ways are out of scope62for this document. Here we only discuss PI.63 64PI is where a process inherits the priority of another process if the other65process blocks on a lock owned by the current process. To make this easier66to understand, let's use the previous example, with processes A, B, and C again.67 68This time, when A blocks on the lock owned by C, C would inherit the priority69of A. So now if B becomes runnable, it would not preempt C, since C now has70the high priority of A. As soon as C releases the lock, it loses its71inherited priority, and A then can continue with the resource that C had.72 73Terminology74-----------75 76Here I explain some terminology that is used in this document to help describe77the design that is used to implement PI.78 79PI chain80 - The PI chain is an ordered series of locks and processes that cause81 processes to inherit priorities from a previous process that is82 blocked on one of its locks. This is described in more detail83 later in this document.84 85mutex86 - In this document, to differentiate from locks that implement87 PI and spin locks that are used in the PI code, from now on88 the PI locks will be called a mutex.89 90lock91 - In this document from now on, I will use the term lock when92 referring to spin locks that are used to protect parts of the PI93 algorithm. These locks disable preemption for UP (when94 CONFIG_PREEMPT is enabled) and on SMP prevents multiple CPUs from95 entering critical sections simultaneously.96 97spin lock98 - Same as lock above.99 100waiter101 - A waiter is a struct that is stored on the stack of a blocked102 process. Since the scope of the waiter is within the code for103 a process being blocked on the mutex, it is fine to allocate104 the waiter on the process's stack (local variable). This105 structure holds a pointer to the task, as well as the mutex that106 the task is blocked on. It also has rbtree node structures to107 place the task in the waiters rbtree of a mutex as well as the108 pi_waiters rbtree of a mutex owner task (described below).109 110 waiter is sometimes used in reference to the task that is waiting111 on a mutex. This is the same as waiter->task.112 113waiters114 - A list of processes that are blocked on a mutex.115 116top waiter117 - The highest priority process waiting on a specific mutex.118 119top pi waiter120 - The highest priority process waiting on one of the mutexes121 that a specific process owns.122 123Note:124 task and process are used interchangeably in this document, mostly to125 differentiate between two processes that are being described together.126 127 128PI chain129--------130 131The PI chain is a list of processes and mutexes that may cause priority132inheritance to take place. Multiple chains may converge, but a chain133would never diverge, since a process can't be blocked on more than one134mutex at a time.135 136Example::137 138 Process: A, B, C, D, E139 Mutexes: L1, L2, L3, L4140 141 A owns: L1142 B blocked on L1143 B owns L2144 C blocked on L2145 C owns L3146 D blocked on L3147 D owns L4148 E blocked on L4149 150The chain would be::151 152 E->L4->D->L3->C->L2->B->L1->A153 154To show where two chains merge, we could add another process F and155another mutex L5 where B owns L5 and F is blocked on mutex L5.156 157The chain for F would be::158 159 F->L5->B->L1->A160 161Since a process may own more than one mutex, but never be blocked on more than162one, the chains merge.163 164Here we show both chains::165 166 E->L4->D->L3->C->L2-+167 |168 +->B->L1->A169 |170 F->L5-+171 172For PI to work, the processes at the right end of these chains (or we may173also call it the Top of the chain) must be equal to or higher in priority174than the processes to the left or below in the chain.175 176Also since a mutex may have more than one process blocked on it, we can177have multiple chains merge at mutexes. If we add another process G that is178blocked on mutex L2::179 180 G->L2->B->L1->A181 182And once again, to show how this can grow I will show the merging chains183again::184 185 E->L4->D->L3->C-+186 +->L2-+187 | |188 G-+ +->B->L1->A189 |190 F->L5-+191 192If process G has the highest priority in the chain, then all the tasks up193the chain (A and B in this example), must have their priorities increased194to that of G.195 196Mutex Waiters Tree197------------------198 199Every mutex keeps track of all the waiters that are blocked on itself. The200mutex has a rbtree to store these waiters by priority. This tree is protected201by a spin lock that is located in the struct of the mutex. This lock is called202wait_lock.203 204 205Task PI Tree206------------207 208To keep track of the PI chains, each process has its own PI rbtree. This is209a tree of all top waiters of the mutexes that are owned by the process.210Note that this tree only holds the top waiters and not all waiters that are211blocked on mutexes owned by the process.212 213The top of the task's PI tree is always the highest priority task that214is waiting on a mutex that is owned by the task. So if the task has215inherited a priority, it will always be the priority of the task that is216at the top of this tree.217 218This tree is stored in the task structure of a process as a rbtree called219pi_waiters. It is protected by a spin lock also in the task structure,220called pi_lock. This lock may also be taken in interrupt context, so when221locking the pi_lock, interrupts must be disabled.222 223 224Depth of the PI Chain225---------------------226 227The maximum depth of the PI chain is not dynamic, and could actually be228defined. But is very complex to figure it out, since it depends on all229the nesting of mutexes. Let's look at the example where we have 3 mutexes,230L1, L2, and L3, and four separate functions func1, func2, func3 and func4.231The following shows a locking order of L1->L2->L3, but may not actually232be directly nested that way::233 234 void func1(void)235 {236 mutex_lock(L1);237 238 /* do anything */239 240 mutex_unlock(L1);241 }242 243 void func2(void)244 {245 mutex_lock(L1);246 mutex_lock(L2);247 248 /* do something */249 250 mutex_unlock(L2);251 mutex_unlock(L1);252 }253 254 void func3(void)255 {256 mutex_lock(L2);257 mutex_lock(L3);258 259 /* do something else */260 261 mutex_unlock(L3);262 mutex_unlock(L2);263 }264 265 void func4(void)266 {267 mutex_lock(L3);268 269 /* do something again */270 271 mutex_unlock(L3);272 }273 274Now we add 4 processes that run each of these functions separately.275Processes A, B, C, and D which run functions func1, func2, func3 and func4276respectively, and such that D runs first and A last. With D being preempted277in func4 in the "do something again" area, we have a locking that follows::278 279 D owns L3280 C blocked on L3281 C owns L2282 B blocked on L2283 B owns L1284 A blocked on L1285 286 And thus we have the chain A->L1->B->L2->C->L3->D.287 288This gives us a PI depth of 4 (four processes), but looking at any of the289functions individually, it seems as though they only have at most a locking290depth of two. So, although the locking depth is defined at compile time,291it still is very difficult to find the possibilities of that depth.292 293Now since mutexes can be defined by user-land applications, we don't want a DOS294type of application that nests large amounts of mutexes to create a large295PI chain, and have the code holding spin locks while looking at a large296amount of data. So to prevent this, the implementation not only implements297a maximum lock depth, but also only holds at most two different locks at a298time, as it walks the PI chain. More about this below.299 300 301Mutex owner and flags302---------------------303 304The mutex structure contains a pointer to the owner of the mutex. If the305mutex is not owned, this owner is set to NULL. Since all architectures306have the task structure on at least a two byte alignment (and if this is307not true, the rtmutex.c code will be broken!), this allows for the least308significant bit to be used as a flag. Bit 0 is used as the "Has Waiters"309flag. It's set whenever there are waiters on a mutex.310 311See Documentation/locking/rt-mutex.rst for further details.312 313cmpxchg Tricks314--------------315 316Some architectures implement an atomic cmpxchg (Compare and Exchange). This317is used (when applicable) to keep the fast path of grabbing and releasing318mutexes short.319 320cmpxchg is basically the following function performed atomically::321 322 unsigned long _cmpxchg(unsigned long *A, unsigned long *B, unsigned long *C)323 {324 unsigned long T = *A;325 if (*A == *B) {326 *A = *C;327 }328 return T;329 }330 #define cmpxchg(a,b,c) _cmpxchg(&a,&b,&c)331 332This is really nice to have, since it allows you to only update a variable333if the variable is what you expect it to be. You know if it succeeded if334the return value (the old value of A) is equal to B.335 336The macro rt_mutex_cmpxchg is used to try to lock and unlock mutexes. If337the architecture does not support CMPXCHG, then this macro is simply set338to fail every time. But if CMPXCHG is supported, then this will339help out extremely to keep the fast path short.340 341The use of rt_mutex_cmpxchg with the flags in the owner field help optimize342the system for architectures that support it. This will also be explained343later in this document.344 345 346Priority adjustments347--------------------348 349The implementation of the PI code in rtmutex.c has several places that a350process must adjust its priority. With the help of the pi_waiters of a351process this is rather easy to know what needs to be adjusted.352 353The functions implementing the task adjustments are rt_mutex_adjust_prio354and rt_mutex_setprio. rt_mutex_setprio is only used in rt_mutex_adjust_prio.355 356rt_mutex_adjust_prio examines the priority of the task, and the highest357priority process that is waiting any of mutexes owned by the task. Since358the pi_waiters of a task holds an order by priority of all the top waiters359of all the mutexes that the task owns, we simply need to compare the top360pi waiter to its own normal/deadline priority and take the higher one.361Then rt_mutex_setprio is called to adjust the priority of the task to the362new priority. Note that rt_mutex_setprio is defined in kernel/sched/core.c363to implement the actual change in priority.364 365Note:366 For the "prio" field in task_struct, the lower the number, the367 higher the priority. A "prio" of 5 is of higher priority than a368 "prio" of 10.369 370It is interesting to note that rt_mutex_adjust_prio can either increase371or decrease the priority of the task. In the case that a higher priority372process has just blocked on a mutex owned by the task, rt_mutex_adjust_prio373would increase/boost the task's priority. But if a higher priority task374were for some reason to leave the mutex (timeout or signal), this same function375would decrease/unboost the priority of the task. That is because the pi_waiters376always contains the highest priority task that is waiting on a mutex owned377by the task, so we only need to compare the priority of that top pi waiter378to the normal priority of the given task.379 380 381High level overview of the PI chain walk382----------------------------------------383 384The PI chain walk is implemented by the function rt_mutex_adjust_prio_chain.385 386The implementation has gone through several iterations, and has ended up387with what we believe is the best. It walks the PI chain by only grabbing388at most two locks at a time, and is very efficient.389 390The rt_mutex_adjust_prio_chain can be used either to boost or lower process391priorities.392 393rt_mutex_adjust_prio_chain is called with a task to be checked for PI394(de)boosting (the owner of a mutex that a process is blocking on), a flag to395check for deadlocking, the mutex that the task owns, a pointer to a waiter396that is the process's waiter struct that is blocked on the mutex (although this397parameter may be NULL for deboosting), a pointer to the mutex on which the task398is blocked, and a top_task as the top waiter of the mutex.399 400For this explanation, I will not mention deadlock detection. This explanation401will try to stay at a high level.402 403When this function is called, there are no locks held. That also means404that the state of the owner and lock can change when entered into this function.405 406Before this function is called, the task has already had rt_mutex_adjust_prio407performed on it. This means that the task is set to the priority that it408should be at, but the rbtree nodes of the task's waiter have not been updated409with the new priorities, and this task may not be in the proper locations410in the pi_waiters and waiters trees that the task is blocked on. This function411solves all that.412 413The main operation of this function is summarized by Thomas Gleixner in414rtmutex.c. See the 'Chain walk basics and protection scope' comment for further415details.416 417Taking of a mutex (The walk through)418------------------------------------419 420OK, now let's take a look at the detailed walk through of what happens when421taking a mutex.422 423The first thing that is tried is the fast taking of the mutex. This is424done when we have CMPXCHG enabled (otherwise the fast taking automatically425fails). Only when the owner field of the mutex is NULL can the lock be426taken with the CMPXCHG and nothing else needs to be done.427 428If there is contention on the lock, we go about the slow path429(rt_mutex_slowlock).430 431The slow path function is where the task's waiter structure is created on432the stack. This is because the waiter structure is only needed for the433scope of this function. The waiter structure holds the nodes to store434the task on the waiters tree of the mutex, and if need be, the pi_waiters435tree of the owner.436 437The wait_lock of the mutex is taken since the slow path of unlocking the438mutex also takes this lock.439 440We then call try_to_take_rt_mutex. This is where the architecture that441does not implement CMPXCHG would always grab the lock (if there's no442contention).443 444try_to_take_rt_mutex is used every time the task tries to grab a mutex in the445slow path. The first thing that is done here is an atomic setting of446the "Has Waiters" flag of the mutex's owner field. By setting this flag447now, the current owner of the mutex being contended for can't release the mutex448without going into the slow unlock path, and it would then need to grab the449wait_lock, which this code currently holds. So setting the "Has Waiters" flag450forces the current owner to synchronize with this code.451 452The lock is taken if the following are true:453 454 1) The lock has no owner455 2) The current task is the highest priority against all other456 waiters of the lock457 458If the task succeeds to acquire the lock, then the task is set as the459owner of the lock, and if the lock still has waiters, the top_waiter460(highest priority task waiting on the lock) is added to this task's461pi_waiters tree.462 463If the lock is not taken by try_to_take_rt_mutex(), then the464task_blocks_on_rt_mutex() function is called. This will add the task to465the lock's waiter tree and propagate the pi chain of the lock as well466as the lock's owner's pi_waiters tree. This is described in the next467section.468 469Task blocks on mutex470--------------------471 472The accounting of a mutex and process is done with the waiter structure of473the process. The "task" field is set to the process, and the "lock" field474to the mutex. The rbtree node of waiter are initialized to the processes475current priority.476 477Since the wait_lock was taken at the entry of the slow lock, we can safely478add the waiter to the task waiter tree. If the current process is the479highest priority process currently waiting on this mutex, then we remove the480previous top waiter process (if it exists) from the pi_waiters of the owner,481and add the current process to that tree. Since the pi_waiter of the owner482has changed, we call rt_mutex_adjust_prio on the owner to see if the owner483should adjust its priority accordingly.484 485If the owner is also blocked on a lock, and had its pi_waiters changed486(or deadlock checking is on), we unlock the wait_lock of the mutex and go ahead487and run rt_mutex_adjust_prio_chain on the owner, as described earlier.488 489Now all locks are released, and if the current process is still blocked on a490mutex (waiter "task" field is not NULL), then we go to sleep (call schedule).491 492Waking up in the loop493---------------------494 495The task can then wake up for a couple of reasons:496 1) The previous lock owner released the lock, and the task now is top_waiter497 2) we received a signal or timeout498 499In both cases, the task will try again to acquire the lock. If it500does, then it will take itself off the waiters tree and set itself back501to the TASK_RUNNING state.502 503In first case, if the lock was acquired by another task before this task504could get the lock, then it will go back to sleep and wait to be woken again.505 506The second case is only applicable for tasks that are grabbing a mutex507that can wake up before getting the lock, either due to a signal or508a timeout (i.e. rt_mutex_timed_futex_lock()). When woken, it will try to509take the lock again, if it succeeds, then the task will return with the510lock held, otherwise it will return with -EINTR if the task was woken511by a signal, or -ETIMEDOUT if it timed out.512 513 514Unlocking the Mutex515-------------------516 517The unlocking of a mutex also has a fast path for those architectures with518CMPXCHG. Since the taking of a mutex on contention always sets the519"Has Waiters" flag of the mutex's owner, we use this to know if we need to520take the slow path when unlocking the mutex. If the mutex doesn't have any521waiters, the owner field of the mutex would equal the current process and522the mutex can be unlocked by just replacing the owner field with NULL.523 524If the owner field has the "Has Waiters" bit set (or CMPXCHG is not available),525the slow unlock path is taken.526 527The first thing done in the slow unlock path is to take the wait_lock of the528mutex. This synchronizes the locking and unlocking of the mutex.529 530A check is made to see if the mutex has waiters or not. On architectures that531do not have CMPXCHG, this is the location that the owner of the mutex will532determine if a waiter needs to be awoken or not. On architectures that533do have CMPXCHG, that check is done in the fast path, but it is still needed534in the slow path too. If a waiter of a mutex woke up because of a signal535or timeout between the time the owner failed the fast path CMPXCHG check and536the grabbing of the wait_lock, the mutex may not have any waiters, thus the537owner still needs to make this check. If there are no waiters then the mutex538owner field is set to NULL, the wait_lock is released and nothing more is539needed.540 541If there are waiters, then we need to wake one up.542 543On the wake up code, the pi_lock of the current owner is taken. The top544waiter of the lock is found and removed from the waiters tree of the mutex545as well as the pi_waiters tree of the current owner. The "Has Waiters" bit is546marked to prevent lower priority tasks from stealing the lock.547 548Finally we unlock the pi_lock of the pending owner and wake it up.549 550 551Contact552-------553 554For updates on this document, please email Steven Rostedt <rostedt@goodmis.org>555 556 557Credits558-------559 560Author: Steven Rostedt <rostedt@goodmis.org>561 562Updated: Alex Shi <alex.shi@linaro.org> - 7/6/2017563 564Original Reviewers:565 Ingo Molnar, Thomas Gleixner, Thomas Duetsch, and566 Randy Dunlap567 568Update (7/6/2017) Reviewers: Steven Rostedt and Sebastian Siewior569 570Updates571-------572 573This document was originally written for 2.6.17-rc3-mm1574was updated on 4.12575