503 lines · plain
1.. _rcu_dereference_doc:2 3PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()4===============================================================5 6Proper care and feeding of address and data dependencies is critically7important to correct use of things like RCU. To this end, the pointers8returned from the rcu_dereference() family of primitives carry address and9data dependencies. These dependencies extend from the rcu_dereference()10macro's load of the pointer to the later use of that pointer to compute11either the address of a later memory access (representing an address12dependency) or the value written by a later memory access (representing13a data dependency).14 15Most of the time, these dependencies are preserved, permitting you to16freely use values from rcu_dereference(). For example, dereferencing17(prefix "*"), field selection ("->"), assignment ("="), address-of18("&"), casts, and addition or subtraction of constants all work quite19naturally and safely. However, because current compilers do not take20either address or data dependencies into account it is still possible21to get into trouble.22 23Follow these rules to preserve the address and data dependencies emanating24from your calls to rcu_dereference() and friends, thus keeping your RCU25readers working properly:26 27- You must use one of the rcu_dereference() family of primitives28 to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU29 will complain. Worse yet, your code can see random memory-corruption30 bugs due to games that compilers and DEC Alpha can play.31 Without one of the rcu_dereference() primitives, compilers32 can reload the value, and won't your code have fun with two33 different values for a single pointer! Without rcu_dereference(),34 DEC Alpha can load a pointer, dereference that pointer, and35 return data preceding initialization that preceded the store36 of the pointer. (As noted later, in recent kernels READ_ONCE()37 also prevents DEC Alpha from playing these tricks.)38 39 In addition, the volatile cast in rcu_dereference() prevents the40 compiler from deducing the resulting pointer value. Please see41 the section entitled "EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH"42 for an example where the compiler can in fact deduce the exact43 value of the pointer, and thus cause misordering.44 45- In the special case where data is added but is never removed46 while readers are accessing the structure, READ_ONCE() may be used47 instead of rcu_dereference(). In this case, use of READ_ONCE()48 takes on the role of the lockless_dereference() primitive that49 was removed in v4.15.50 51- You are only permitted to use rcu_dereference() on pointer values.52 The compiler simply knows too much about integral values to53 trust it to carry dependencies through integer operations.54 There are a very few exceptions, namely that you can temporarily55 cast the pointer to uintptr_t in order to:56 57 - Set bits and clear bits down in the must-be-zero low-order58 bits of that pointer. This clearly means that the pointer59 must have alignment constraints, for example, this does60 *not* work in general for char* pointers.61 62 - XOR bits to translate pointers, as is done in some63 classic buddy-allocator algorithms.64 65 It is important to cast the value back to pointer before66 doing much of anything else with it.67 68- Avoid cancellation when using the "+" and "-" infix arithmetic69 operators. For example, for a given variable "x", avoid70 "(x-(uintptr_t)x)" for char* pointers. The compiler is within its71 rights to substitute zero for this sort of expression, so that72 subsequent accesses no longer depend on the rcu_dereference(),73 again possibly resulting in bugs due to misordering.74 75 Of course, if "p" is a pointer from rcu_dereference(), and "a"76 and "b" are integers that happen to be equal, the expression77 "p+a-b" is safe because its value still necessarily depends on78 the rcu_dereference(), thus maintaining proper ordering.79 80- If you are using RCU to protect JITed functions, so that the81 "()" function-invocation operator is applied to a value obtained82 (directly or indirectly) from rcu_dereference(), you may need to83 interact directly with the hardware to flush instruction caches.84 This issue arises on some systems when a newly JITed function is85 using the same memory that was used by an earlier JITed function.86 87- Do not use the results from relational operators ("==", "!=",88 ">", ">=", "<", or "<=") when dereferencing. For example,89 the following (quite strange) code is buggy::90 91 int *p;92 int *q;93 94 ...95 96 p = rcu_dereference(gp)97 q = &global_q;98 q += p > &oom_p;99 r1 = *q; /* BUGGY!!! */100 101 As before, the reason this is buggy is that relational operators102 are often compiled using branches. And as before, although103 weak-memory machines such as ARM or PowerPC do order stores104 after such branches, but can speculate loads, which can again105 result in misordering bugs.106 107- Be very careful about comparing pointers obtained from108 rcu_dereference() against non-NULL values. As Linus Torvalds109 explained, if the two pointers are equal, the compiler could110 substitute the pointer you are comparing against for the pointer111 obtained from rcu_dereference(). For example::112 113 p = rcu_dereference(gp);114 if (p == &default_struct)115 do_default(p->a);116 117 Because the compiler now knows that the value of "p" is exactly118 the address of the variable "default_struct", it is free to119 transform this code into the following::120 121 p = rcu_dereference(gp);122 if (p == &default_struct)123 do_default(default_struct.a);124 125 On ARM and Power hardware, the load from "default_struct.a"126 can now be speculated, such that it might happen before the127 rcu_dereference(). This could result in bugs due to misordering.128 129 However, comparisons are OK in the following cases:130 131 - The comparison was against the NULL pointer. If the132 compiler knows that the pointer is NULL, you had better133 not be dereferencing it anyway. If the comparison is134 non-equal, the compiler is none the wiser. Therefore,135 it is safe to compare pointers from rcu_dereference()136 against NULL pointers.137 138 - The pointer is never dereferenced after being compared.139 Since there are no subsequent dereferences, the compiler140 cannot use anything it learned from the comparison141 to reorder the non-existent subsequent dereferences.142 This sort of comparison occurs frequently when scanning143 RCU-protected circular linked lists.144 145 Note that if the pointer comparison is done outside146 of an RCU read-side critical section, and the pointer147 is never dereferenced, rcu_access_pointer() should be148 used in place of rcu_dereference(). In most cases,149 it is best to avoid accidental dereferences by testing150 the rcu_access_pointer() return value directly, without151 assigning it to a variable.152 153 Within an RCU read-side critical section, there is little154 reason to use rcu_access_pointer().155 156 - The comparison is against a pointer that references memory157 that was initialized "a long time ago." The reason158 this is safe is that even if misordering occurs, the159 misordering will not affect the accesses that follow160 the comparison. So exactly how long ago is "a long161 time ago"? Here are some possibilities:162 163 - Compile time.164 165 - Boot time.166 167 - Module-init time for module code.168 169 - Prior to kthread creation for kthread code.170 171 - During some prior acquisition of the lock that172 we now hold.173 174 - Before mod_timer() time for a timer handler.175 176 There are many other possibilities involving the Linux177 kernel's wide array of primitives that cause code to178 be invoked at a later time.179 180 - The pointer being compared against also came from181 rcu_dereference(). In this case, both pointers depend182 on one rcu_dereference() or another, so you get proper183 ordering either way.184 185 That said, this situation can make certain RCU usage186 bugs more likely to happen. Which can be a good thing,187 at least if they happen during testing. An example188 of such an RCU usage bug is shown in the section titled189 "EXAMPLE OF AMPLIFIED RCU-USAGE BUG".190 191 - All of the accesses following the comparison are stores,192 so that a control dependency preserves the needed ordering.193 That said, it is easy to get control dependencies wrong.194 Please see the "CONTROL DEPENDENCIES" section of195 Documentation/memory-barriers.txt for more details.196 197 - The pointers are not equal *and* the compiler does198 not have enough information to deduce the value of the199 pointer. Note that the volatile cast in rcu_dereference()200 will normally prevent the compiler from knowing too much.201 202 However, please note that if the compiler knows that the203 pointer takes on only one of two values, a not-equal204 comparison will provide exactly the information that the205 compiler needs to deduce the value of the pointer.206 207- Disable any value-speculation optimizations that your compiler208 might provide, especially if you are making use of feedback-based209 optimizations that take data collected from prior runs. Such210 value-speculation optimizations reorder operations by design.211 212 There is one exception to this rule: Value-speculation213 optimizations that leverage the branch-prediction hardware are214 safe on strongly ordered systems (such as x86), but not on weakly215 ordered systems (such as ARM or Power). Choose your compiler216 command-line options wisely!217 218 219EXAMPLE OF AMPLIFIED RCU-USAGE BUG220----------------------------------221 222Because updaters can run concurrently with RCU readers, RCU readers can223see stale and/or inconsistent values. If RCU readers need fresh or224consistent values, which they sometimes do, they need to take proper225precautions. To see this, consider the following code fragment::226 227 struct foo {228 int a;229 int b;230 int c;231 };232 struct foo *gp1;233 struct foo *gp2;234 235 void updater(void)236 {237 struct foo *p;238 239 p = kmalloc(...);240 if (p == NULL)241 deal_with_it();242 p->a = 42; /* Each field in its own cache line. */243 p->b = 43;244 p->c = 44;245 rcu_assign_pointer(gp1, p);246 p->b = 143;247 p->c = 144;248 rcu_assign_pointer(gp2, p);249 }250 251 void reader(void)252 {253 struct foo *p;254 struct foo *q;255 int r1, r2;256 257 rcu_read_lock();258 p = rcu_dereference(gp2);259 if (p == NULL)260 return;261 r1 = p->b; /* Guaranteed to get 143. */262 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */263 if (p == q) {264 /* The compiler decides that q->c is same as p->c. */265 r2 = p->c; /* Could get 44 on weakly order system. */266 } else {267 r2 = p->c - r1; /* Unconditional access to p->c. */268 }269 rcu_read_unlock();270 do_something_with(r1, r2);271 }272 273You might be surprised that the outcome (r1 == 143 && r2 == 44) is possible,274but you should not be. After all, the updater might have been invoked275a second time between the time reader() loaded into "r1" and the time276that it loaded into "r2". The fact that this same result can occur due277to some reordering from the compiler and CPUs is beside the point.278 279But suppose that the reader needs a consistent view?280 281Then one approach is to use locking, for example, as follows::282 283 struct foo {284 int a;285 int b;286 int c;287 spinlock_t lock;288 };289 struct foo *gp1;290 struct foo *gp2;291 292 void updater(void)293 {294 struct foo *p;295 296 p = kmalloc(...);297 if (p == NULL)298 deal_with_it();299 spin_lock(&p->lock);300 p->a = 42; /* Each field in its own cache line. */301 p->b = 43;302 p->c = 44;303 spin_unlock(&p->lock);304 rcu_assign_pointer(gp1, p);305 spin_lock(&p->lock);306 p->b = 143;307 p->c = 144;308 spin_unlock(&p->lock);309 rcu_assign_pointer(gp2, p);310 }311 312 void reader(void)313 {314 struct foo *p;315 struct foo *q;316 int r1, r2;317 318 rcu_read_lock();319 p = rcu_dereference(gp2);320 if (p == NULL)321 return;322 spin_lock(&p->lock);323 r1 = p->b; /* Guaranteed to get 143. */324 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */325 if (p == q) {326 /* The compiler decides that q->c is same as p->c. */327 r2 = p->c; /* Locking guarantees r2 == 144. */328 } else {329 spin_lock(&q->lock);330 r2 = q->c - r1;331 spin_unlock(&q->lock);332 }333 rcu_read_unlock();334 spin_unlock(&p->lock);335 do_something_with(r1, r2);336 }337 338As always, use the right tool for the job!339 340 341EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH342-----------------------------------------343 344If a pointer obtained from rcu_dereference() compares not-equal to some345other pointer, the compiler normally has no clue what the value of the346first pointer might be. This lack of knowledge prevents the compiler347from carrying out optimizations that otherwise might destroy the ordering348guarantees that RCU depends on. And the volatile cast in rcu_dereference()349should prevent the compiler from guessing the value.350 351But without rcu_dereference(), the compiler knows more than you might352expect. Consider the following code fragment::353 354 struct foo {355 int a;356 int b;357 };358 static struct foo variable1;359 static struct foo variable2;360 static struct foo *gp = &variable1;361 362 void updater(void)363 {364 initialize_foo(&variable2);365 rcu_assign_pointer(gp, &variable2);366 /*367 * The above is the only store to gp in this translation unit,368 * and the address of gp is not exported in any way.369 */370 }371 372 int reader(void)373 {374 struct foo *p;375 376 p = gp;377 barrier();378 if (p == &variable1)379 return p->a; /* Must be variable1.a. */380 else381 return p->b; /* Must be variable2.b. */382 }383 384Because the compiler can see all stores to "gp", it knows that the only385possible values of "gp" are "variable1" on the one hand and "variable2"386on the other. The comparison in reader() therefore tells the compiler387the exact value of "p" even in the not-equals case. This allows the388compiler to make the return values independent of the load from "gp",389in turn destroying the ordering between this load and the loads of the390return values. This can result in "p->b" returning pre-initialization391garbage values on weakly ordered systems.392 393In short, rcu_dereference() is *not* optional when you are going to394dereference the resulting pointer.395 396 397WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE?398------------------------------------------------------------399 400First, please avoid using rcu_dereference_raw() and also please avoid401using rcu_dereference_check() and rcu_dereference_protected() with a402second argument with a constant value of 1 (or true, for that matter).403With that caution out of the way, here is some guidance for which404member of the rcu_dereference() to use in various situations:405 4061. If the access needs to be within an RCU read-side critical407 section, use rcu_dereference(). With the new consolidated408 RCU flavors, an RCU read-side critical section is entered409 using rcu_read_lock(), anything that disables bottom halves,410 anything that disables interrupts, or anything that disables411 preemption. Please note that spinlock critical sections412 are also implied RCU read-side critical sections, even when413 they are preemptible, as they are in kernels built with414 CONFIG_PREEMPT_RT=y.415 4162. If the access might be within an RCU read-side critical section417 on the one hand, or protected by (say) my_lock on the other,418 use rcu_dereference_check(), for example::419 420 p1 = rcu_dereference_check(p->rcu_protected_pointer,421 lockdep_is_held(&my_lock));422 423 4243. If the access might be within an RCU read-side critical section425 on the one hand, or protected by either my_lock or your_lock on426 the other, again use rcu_dereference_check(), for example::427 428 p1 = rcu_dereference_check(p->rcu_protected_pointer,429 lockdep_is_held(&my_lock) ||430 lockdep_is_held(&your_lock));431 4324. If the access is on the update side, so that it is always protected433 by my_lock, use rcu_dereference_protected()::434 435 p1 = rcu_dereference_protected(p->rcu_protected_pointer,436 lockdep_is_held(&my_lock));437 438 This can be extended to handle multiple locks as in #3 above,439 and both can be extended to check other conditions as well.440 4415. If the protection is supplied by the caller, and is thus unknown442 to this code, that is the rare case when rcu_dereference_raw()443 is appropriate. In addition, rcu_dereference_raw() might be444 appropriate when the lockdep expression would be excessively445 complex, except that a better approach in that case might be to446 take a long hard look at your synchronization design. Still,447 there are data-locking cases where any one of a very large number448 of locks or reference counters suffices to protect the pointer,449 so rcu_dereference_raw() does have its place.450 451 However, its place is probably quite a bit smaller than one452 might expect given the number of uses in the current kernel.453 Ditto for its synonym, rcu_dereference_check( ... , 1), and454 its close relative, rcu_dereference_protected(... , 1).455 456 457SPARSE CHECKING OF RCU-PROTECTED POINTERS458-----------------------------------------459 460The sparse static-analysis tool checks for non-RCU access to RCU-protected461pointers, which can result in "interesting" bugs due to compiler462optimizations involving invented loads and perhaps also load tearing.463For example, suppose someone mistakenly does something like this::464 465 p = q->rcu_protected_pointer;466 do_something_with(p->a);467 do_something_else_with(p->b);468 469If register pressure is high, the compiler might optimize "p" out470of existence, transforming the code to something like this::471 472 do_something_with(q->rcu_protected_pointer->a);473 do_something_else_with(q->rcu_protected_pointer->b);474 475This could fatally disappoint your code if q->rcu_protected_pointer476changed in the meantime. Nor is this a theoretical problem: Exactly477this sort of bug cost Paul E. McKenney (and several of his innocent478colleagues) a three-day weekend back in the early 1990s.479 480Load tearing could of course result in dereferencing a mashup of a pair481of pointers, which also might fatally disappoint your code.482 483These problems could have been avoided simply by making the code instead484read as follows::485 486 p = rcu_dereference(q->rcu_protected_pointer);487 do_something_with(p->a);488 do_something_else_with(p->b);489 490Unfortunately, these sorts of bugs can be extremely hard to spot during491review. This is where the sparse tool comes into play, along with the492"__rcu" marker. If you mark a pointer declaration, whether in a structure493or as a formal parameter, with "__rcu", which tells sparse to complain if494this pointer is accessed directly. It will also cause sparse to complain495if a pointer not marked with "__rcu" is accessed using rcu_dereference()496and friends. For example, ->rcu_protected_pointer might be declared as497follows::498 499 struct foo __rcu *rcu_protected_pointer;500 501Use of "__rcu" is opt-in. If you choose not to use it, then you should502ignore the sparse warnings.503