762 lines · plain
1=========================2CPU hotplug in the Kernel3=========================4 5:Date: September, 20216:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,7 Rusty Russell <rusty@rustcorp.com.au>,8 Srivatsa Vaddagiri <vatsa@in.ibm.com>,9 Ashok Raj <ashok.raj@intel.com>,10 Joel Schopp <jschopp@austin.ibm.com>,11 Thomas Gleixner <tglx@linutronix.de>12 13Introduction14============15 16Modern advances in system architectures have introduced advanced error17reporting and correction capabilities in processors. There are couple OEMS that18support NUMA hardware which are hot pluggable as well, where physical node19insertion and removal require support for CPU hotplug.20 21Such advances require CPUs available to a kernel to be removed either for22provisioning reasons, or for RAS purposes to keep an offending CPU off23system execution path. Hence the need for CPU hotplug support in the24Linux kernel.25 26A more novel use of CPU-hotplug support is its use today in suspend resume27support for SMP. Dual-core and HT support makes even a laptop run SMP kernels28which didn't support these methods.29 30 31Command Line Switches32=====================33``maxcpus=n``34 Restrict boot time CPUs to *n*. Say if you have four CPUs, using35 ``maxcpus=2`` will only boot two. You can choose to bring the36 other CPUs later online.37 38``nr_cpus=n``39 Restrict the total amount of CPUs the kernel will support. If the number40 supplied here is lower than the number of physically available CPUs, then41 those CPUs can not be brought online later.42 43``possible_cpus=n``44 This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.45 46 This option is limited to the X86 and S390 architecture.47 48``cpu0_hotplug``49 Allow to shutdown CPU0.50 51 This option is limited to the X86 architecture.52 53CPU maps54========55 56``cpu_possible_mask``57 Bitmap of possible CPUs that can ever be available in the58 system. This is used to allocate some boot time memory for per_cpu variables59 that aren't designed to grow/shrink as CPUs are made available or removed.60 Once set during boot time discovery phase, the map is static, i.e no bits61 are added or removed anytime. Trimming it accurately for your system needs62 upfront can save some boot time memory.63 64``cpu_online_mask``65 Bitmap of all CPUs currently online. Its set in ``__cpu_up()``66 after a CPU is available for kernel scheduling and ready to receive67 interrupts from devices. Its cleared when a CPU is brought down using68 ``__cpu_disable()``, before which all OS services including interrupts are69 migrated to another target CPU.70 71``cpu_present_mask``72 Bitmap of CPUs currently present in the system. Not all73 of them may be online. When physical hotplug is processed by the relevant74 subsystem (e.g ACPI) can change and new bit either be added or removed75 from the map depending on the event is hot-add/hot-remove. There are currently76 no locking rules as of now. Typical usage is to init topology during boot,77 at which time hotplug is disabled.78 79You really don't need to manipulate any of the system CPU maps. They should80be read-only for most use. When setting up per-cpu resources almost always use81``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro82``for_each_cpu()`` can be used to iterate over a custom CPU mask.83 84Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.85 86 87Using CPU hotplug88=================89 90The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently91available on multiple architectures including ARM, MIPS, PowerPC and X86. The92configuration is done via the sysfs interface::93 94 $ ls -lh /sys/devices/system/cpu95 total 096 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu097 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu198 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu299 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu3100 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu4101 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu5102 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu6103 drwxr-xr-x 9 root root 0 Dec 21 16:33 cpu7104 drwxr-xr-x 2 root root 0 Dec 21 16:33 hotplug105 -r--r--r-- 1 root root 4.0K Dec 21 16:33 offline106 -r--r--r-- 1 root root 4.0K Dec 21 16:33 online107 -r--r--r-- 1 root root 4.0K Dec 21 16:33 possible108 -r--r--r-- 1 root root 4.0K Dec 21 16:33 present109 110The files *offline*, *online*, *possible*, *present* represent the CPU masks.111Each CPU folder contains an *online* file which controls the logical on (1) and112off (0) state. To logically shutdown CPU4::113 114 $ echo 0 > /sys/devices/system/cpu/cpu4/online115 smpboot: CPU 4 is now offline116 117Once the CPU is shutdown, it will be removed from */proc/interrupts*,118*/proc/cpuinfo* and should also not be shown visible by the *top* command. To119bring CPU4 back online::120 121 $ echo 1 > /sys/devices/system/cpu/cpu4/online122 smpboot: Booting Node 0 Processor 4 APIC 0x1123 124The CPU is usable again. This should work on all CPUs, but CPU0 is often special125and excluded from CPU hotplug.126 127The CPU hotplug coordination128============================129 130The offline case131----------------132 133Once a CPU has been logically shutdown the teardown callbacks of registered134hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating135at state ``CPUHP_OFFLINE``. This includes:136 137* If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*138 will be set to true.139* All processes are migrated away from this outgoing CPU to new CPUs.140 The new CPU is chosen from each process' current cpuset, which may be141 a subset of all online CPUs.142* All interrupts targeted to this CPU are migrated to a new CPU143* timers are also migrated to a new CPU144* Once all services are migrated, kernel calls an arch specific routine145 ``__cpu_disable()`` to perform arch specific cleanup.146 147 148The CPU hotplug API149===================150 151CPU hotplug state machine152-------------------------153 154CPU hotplug uses a trivial state machine with a linear state space from155CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown156callback.157 158When a CPU is onlined, the startup callbacks are invoked sequentially until159the state CPUHP_ONLINE is reached. They can also be invoked when the160callbacks of a state are set up or an instance is added to a multi-instance161state.162 163When a CPU is offlined the teardown callbacks are invoked in the reverse164order sequentially until the state CPUHP_OFFLINE is reached. They can also165be invoked when the callbacks of a state are removed or an instance is166removed from a multi-instance state.167 168If a usage site requires only a callback in one direction of the hotplug169operations (CPU online or CPU offline) then the other not-required callback170can be set to NULL when the state is set up.171 172The state space is divided into three sections:173 174* The PREPARE section175 176 The PREPARE section covers the state space from CPUHP_OFFLINE to177 CPUHP_BRINGUP_CPU.178 179 The startup callbacks in this section are invoked before the CPU is180 started during a CPU online operation. The teardown callbacks are invoked181 after the CPU has become dysfunctional during a CPU offline operation.182 183 The callbacks are invoked on a control CPU as they can't obviously run on184 the hotplugged CPU which is either not yet started or has become185 dysfunctional already.186 187 The startup callbacks are used to setup resources which are required to188 bring a CPU successfully online. The teardown callbacks are used to free189 resources or to move pending work to an online CPU after the hotplugged190 CPU became dysfunctional.191 192 The startup callbacks are allowed to fail. If a callback fails, the CPU193 online operation is aborted and the CPU is brought down to the previous194 state (usually CPUHP_OFFLINE) again.195 196 The teardown callbacks in this section are not allowed to fail.197 198* The STARTING section199 200 The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1201 and CPUHP_AP_ONLINE.202 203 The startup callbacks in this section are invoked on the hotplugged CPU204 with interrupts disabled during a CPU online operation in the early CPU205 setup code. The teardown callbacks are invoked with interrupts disabled206 on the hotplugged CPU during a CPU offline operation shortly before the207 CPU is completely shut down.208 209 The callbacks in this section are not allowed to fail.210 211 The callbacks are used for low level hardware initialization/shutdown and212 for core subsystems.213 214* The ONLINE section215 216 The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and217 CPUHP_ONLINE.218 219 The startup callbacks in this section are invoked on the hotplugged CPU220 during a CPU online operation. The teardown callbacks are invoked on the221 hotplugged CPU during a CPU offline operation.222 223 The callbacks are invoked in the context of the per CPU hotplug thread,224 which is pinned on the hotplugged CPU. The callbacks are invoked with225 interrupts and preemption enabled.226 227 The callbacks are allowed to fail. When a callback fails the hotplug228 operation is aborted and the CPU is brought back to the previous state.229 230CPU online/offline operations231-----------------------------232 233A successful online operation looks like this::234 235 [CPUHP_OFFLINE]236 [CPUHP_OFFLINE + 1]->startup() -> success237 [CPUHP_OFFLINE + 2]->startup() -> success238 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL239 ...240 [CPUHP_BRINGUP_CPU]->startup() -> success241 === End of PREPARE section242 [CPUHP_BRINGUP_CPU + 1]->startup() -> success243 ...244 [CPUHP_AP_ONLINE]->startup() -> success245 === End of STARTUP section246 [CPUHP_AP_ONLINE + 1]->startup() -> success247 ...248 [CPUHP_ONLINE - 1]->startup() -> success249 [CPUHP_ONLINE]250 251A successful offline operation looks like this::252 253 [CPUHP_ONLINE]254 [CPUHP_ONLINE - 1]->teardown() -> success255 ...256 [CPUHP_AP_ONLINE + 1]->teardown() -> success257 === Start of STARTUP section258 [CPUHP_AP_ONLINE]->teardown() -> success259 ...260 [CPUHP_BRINGUP_ONLINE - 1]->teardown()261 ...262 === Start of PREPARE section263 [CPUHP_BRINGUP_CPU]->teardown()264 [CPUHP_OFFLINE + 3]->teardown()265 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL266 [CPUHP_OFFLINE + 1]->teardown()267 [CPUHP_OFFLINE]268 269A failed online operation looks like this::270 271 [CPUHP_OFFLINE]272 [CPUHP_OFFLINE + 1]->startup() -> success273 [CPUHP_OFFLINE + 2]->startup() -> success274 [CPUHP_OFFLINE + 3] -> skipped because startup == NULL275 ...276 [CPUHP_BRINGUP_CPU]->startup() -> success277 === End of PREPARE section278 [CPUHP_BRINGUP_CPU + 1]->startup() -> success279 ...280 [CPUHP_AP_ONLINE]->startup() -> success281 === End of STARTUP section282 [CPUHP_AP_ONLINE + 1]->startup() -> success283 ---284 [CPUHP_AP_ONLINE + N]->startup() -> fail285 [CPUHP_AP_ONLINE + (N - 1)]->teardown()286 ...287 [CPUHP_AP_ONLINE + 1]->teardown()288 === Start of STARTUP section289 [CPUHP_AP_ONLINE]->teardown()290 ...291 [CPUHP_BRINGUP_ONLINE - 1]->teardown()292 ...293 === Start of PREPARE section294 [CPUHP_BRINGUP_CPU]->teardown()295 [CPUHP_OFFLINE + 3]->teardown()296 [CPUHP_OFFLINE + 2] -> skipped because teardown == NULL297 [CPUHP_OFFLINE + 1]->teardown()298 [CPUHP_OFFLINE]299 300A failed offline operation looks like this::301 302 [CPUHP_ONLINE]303 [CPUHP_ONLINE - 1]->teardown() -> success304 ...305 [CPUHP_ONLINE - N]->teardown() -> fail306 [CPUHP_ONLINE - (N - 1)]->startup()307 ...308 [CPUHP_ONLINE - 1]->startup()309 [CPUHP_ONLINE]310 311Recursive failures cannot be handled sensibly. Look at the following312example of a recursive fail due to a failed offline operation: ::313 314 [CPUHP_ONLINE]315 [CPUHP_ONLINE - 1]->teardown() -> success316 ...317 [CPUHP_ONLINE - N]->teardown() -> fail318 [CPUHP_ONLINE - (N - 1)]->startup() -> success319 [CPUHP_ONLINE - (N - 2)]->startup() -> fail320 321The CPU hotplug state machine stops right here and does not try to go back322down again because that would likely result in an endless loop::323 324 [CPUHP_ONLINE - (N - 1)]->teardown() -> success325 [CPUHP_ONLINE - N]->teardown() -> fail326 [CPUHP_ONLINE - (N - 1)]->startup() -> success327 [CPUHP_ONLINE - (N - 2)]->startup() -> fail328 [CPUHP_ONLINE - (N - 1)]->teardown() -> success329 [CPUHP_ONLINE - N]->teardown() -> fail330 331Lather, rinse and repeat. In this case the CPU left in state::332 333 [CPUHP_ONLINE - (N - 1)]334 335which at least lets the system make progress and gives the user a chance to336debug or even resolve the situation.337 338Allocating a state339------------------340 341There are two ways to allocate a CPU hotplug state:342 343* Static allocation344 345 Static allocation has to be used when the subsystem or driver has346 ordering requirements versus other CPU hotplug states. E.g. the PERF core347 startup callback has to be invoked before the PERF driver startup348 callbacks during a CPU online operation. During a CPU offline operation349 the driver teardown callbacks have to be invoked before the core teardown350 callback. The statically allocated states are described by constants in351 the cpuhp_state enum which can be found in include/linux/cpuhotplug.h.352 353 Insert the state into the enum at the proper place so the ordering354 requirements are fulfilled. The state constant has to be used for state355 setup and removal.356 357 Static allocation is also required when the state callbacks are not set358 up at runtime and are part of the initializer of the CPU hotplug state359 array in kernel/cpu.c.360 361* Dynamic allocation362 363 When there are no ordering requirements for the state callbacks then364 dynamic allocation is the preferred method. The state number is allocated365 by the setup function and returned to the caller on success.366 367 Only the PREPARE and ONLINE sections provide a dynamic allocation368 range. The STARTING section does not as most of the callbacks in that369 section have explicit ordering requirements.370 371Setup of a CPU hotplug state372----------------------------373 374The core code provides the following functions to setup a state:375 376* cpuhp_setup_state(state, name, startup, teardown)377* cpuhp_setup_state_nocalls(state, name, startup, teardown)378* cpuhp_setup_state_cpuslocked(state, name, startup, teardown)379* cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown)380 381For cases where a driver or a subsystem has multiple instances and the same382CPU hotplug state callbacks need to be invoked for each instance, the CPU383hotplug core provides multi-instance support. The advantage over driver384specific instance lists is that the instance related functions are fully385serialized against CPU hotplug operations and provide the automatic386invocations of the state callbacks on add and removal. To set up such a387multi-instance state the following function is available:388 389* cpuhp_setup_state_multi(state, name, startup, teardown)390 391The @state argument is either a statically allocated state or one of the392constants for dynamically allocated states - CPUHP_BP_PREPARE_DYN,393CPUHP_AP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for394which a dynamic state should be allocated.395 396The @name argument is used for sysfs output and for instrumentation. The397naming convention is "subsys:mode" or "subsys/driver:mode",398e.g. "perf:mode" or "perf/x86:mode". The common mode names are:399 400======== =======================================================401prepare For states in the PREPARE section402 403dead For states in the PREPARE section which do not provide404 a startup callback405 406starting For states in the STARTING section407 408dying For states in the STARTING section which do not provide409 a startup callback410 411online For states in the ONLINE section412 413offline For states in the ONLINE section which do not provide414 a startup callback415======== =======================================================416 417As the @name argument is only used for sysfs and instrumentation other mode418descriptors can be used as well if they describe the nature of the state419better than the common ones.420 421Examples for @name arguments: "perf/online", "perf/x86:prepare",422"RCU/tree:dying", "sched/waitempty"423 424The @startup argument is a function pointer to the callback which should be425invoked during a CPU online operation. If the usage site does not require a426startup callback set the pointer to NULL.427 428The @teardown argument is a function pointer to the callback which should429be invoked during a CPU offline operation. If the usage site does not430require a teardown callback set the pointer to NULL.431 432The functions differ in the way how the installed callbacks are treated:433 434 * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked()435 and cpuhp_setup_state_multi() only install the callbacks436 437 * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the438 callbacks and invoke the @startup callback (if not NULL) for all online439 CPUs which have currently a state greater than the newly installed440 state. Depending on the state section the callback is either invoked on441 the current CPU (PREPARE section) or on each online CPU (ONLINE442 section) in the context of the CPU's hotplug thread.443 444 If a callback fails for CPU N then the teardown callback for CPU445 0 .. N-1 is invoked to rollback the operation. The state setup fails,446 the callbacks for the state are not installed and in case of dynamic447 allocation the allocated state is freed.448 449The state setup and the callback invocations are serialized against CPU450hotplug operations. If the setup function has to be called from a CPU451hotplug read locked region, then the _cpuslocked() variants have to be452used. These functions cannot be used from within CPU hotplug callbacks.453 454The function return values:455 ======== ===================================================================456 0 Statically allocated state was successfully set up457 458 >0 Dynamically allocated state was successfully set up.459 460 The returned number is the state number which was allocated. If461 the state callbacks have to be removed later, e.g. module462 removal, then this number has to be saved by the caller and used463 as @state argument for the state remove function. For464 multi-instance states the dynamically allocated state number is465 also required as @state argument for the instance add/remove466 operations.467 468 <0 Operation failed469 ======== ===================================================================470 471Removal of a CPU hotplug state472------------------------------473 474To remove a previously set up state, the following functions are provided:475 476* cpuhp_remove_state(state)477* cpuhp_remove_state_nocalls(state)478* cpuhp_remove_state_nocalls_cpuslocked(state)479* cpuhp_remove_multi_state(state)480 481The @state argument is either a statically allocated state or the state482number which was allocated in the dynamic range by cpuhp_setup_state*(). If483the state is in the dynamic range, then the state number is freed and484available for dynamic allocation again.485 486The functions differ in the way how the installed callbacks are treated:487 488 * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked()489 and cpuhp_remove_multi_state() only remove the callbacks.490 491 * cpuhp_remove_state() removes the callbacks and invokes the teardown492 callback (if not NULL) for all online CPUs which have currently a state493 greater than the removed state. Depending on the state section the494 callback is either invoked on the current CPU (PREPARE section) or on495 each online CPU (ONLINE section) in the context of the CPU's hotplug496 thread.497 498 In order to complete the removal, the teardown callback should not fail.499 500The state removal and the callback invocations are serialized against CPU501hotplug operations. If the remove function has to be called from a CPU502hotplug read locked region, then the _cpuslocked() variants have to be503used. These functions cannot be used from within CPU hotplug callbacks.504 505If a multi-instance state is removed then the caller has to remove all506instances first.507 508Multi-Instance state instance management509----------------------------------------510 511Once the multi-instance state is set up, instances can be added to the512state:513 514 * cpuhp_state_add_instance(state, node)515 * cpuhp_state_add_instance_nocalls(state, node)516 517The @state argument is either a statically allocated state or the state518number which was allocated in the dynamic range by cpuhp_setup_state_multi().519 520The @node argument is a pointer to an hlist_node which is embedded in the521instance's data structure. The pointer is handed to the multi-instance522state callbacks and can be used by the callback to retrieve the instance523via container_of().524 525The functions differ in the way how the installed callbacks are treated:526 527 * cpuhp_state_add_instance_nocalls() and only adds the instance to the528 multi-instance state's node list.529 530 * cpuhp_state_add_instance() adds the instance and invokes the startup531 callback (if not NULL) associated with @state for all online CPUs which532 have currently a state greater than @state. The callback is only533 invoked for the to be added instance. Depending on the state section534 the callback is either invoked on the current CPU (PREPARE section) or535 on each online CPU (ONLINE section) in the context of the CPU's hotplug536 thread.537 538 If a callback fails for CPU N then the teardown callback for CPU539 0 .. N-1 is invoked to rollback the operation, the function fails and540 the instance is not added to the node list of the multi-instance state.541 542To remove an instance from the state's node list these functions are543available:544 545 * cpuhp_state_remove_instance(state, node)546 * cpuhp_state_remove_instance_nocalls(state, node)547 548The arguments are the same as for the cpuhp_state_add_instance*()549variants above.550 551The functions differ in the way how the installed callbacks are treated:552 553 * cpuhp_state_remove_instance_nocalls() only removes the instance from the554 state's node list.555 556 * cpuhp_state_remove_instance() removes the instance and invokes the557 teardown callback (if not NULL) associated with @state for all online558 CPUs which have currently a state greater than @state. The callback is559 only invoked for the to be removed instance. Depending on the state560 section the callback is either invoked on the current CPU (PREPARE561 section) or on each online CPU (ONLINE section) in the context of the562 CPU's hotplug thread.563 564 In order to complete the removal, the teardown callback should not fail.565 566The node list add/remove operations and the callback invocations are567serialized against CPU hotplug operations. These functions cannot be used568from within CPU hotplug callbacks and CPU hotplug read locked regions.569 570Examples571--------572 573Setup and teardown a statically allocated state in the STARTING section for574notifications on online and offline operations::575 576 ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying);577 if (ret < 0)578 return ret;579 ....580 cpuhp_remove_state(CPUHP_SUBSYS_STARTING);581 582Setup and teardown a dynamically allocated state in the ONLINE section583for notifications on offline operations::584 585 state = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline);586 if (state < 0)587 return state;588 ....589 cpuhp_remove_state(state);590 591Setup and teardown a dynamically allocated state in the ONLINE section592for notifications on online operations without invoking the callbacks::593 594 state = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL);595 if (state < 0)596 return state;597 ....598 cpuhp_remove_state_nocalls(state);599 600Setup, use and teardown a dynamically allocated multi-instance state in the601ONLINE section for notifications on online and offline operation::602 603 state = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline);604 if (state < 0)605 return state;606 ....607 ret = cpuhp_state_add_instance(state, &inst1->node);608 if (ret)609 return ret;610 ....611 ret = cpuhp_state_add_instance(state, &inst2->node);612 if (ret)613 return ret;614 ....615 cpuhp_remove_instance(state, &inst1->node);616 ....617 cpuhp_remove_instance(state, &inst2->node);618 ....619 remove_multi_state(state);620 621 622Testing of hotplug states623=========================624 625One way to verify whether a custom state is working as expected or not is to626shutdown a CPU and then put it online again. It is also possible to put the CPU627to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to628*CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*629which would lead to rollback to the online state.630 631All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states`` ::632 633 $ tail /sys/devices/system/cpu/hotplug/states634 138: mm/vmscan:online635 139: mm/vmstat:online636 140: lib/percpu_cnt:online637 141: acpi/cpu-drv:online638 142: base/cacheinfo:online639 143: virtio/net:online640 144: x86/mce:online641 145: printk:online642 168: sched:active643 169: online644 645To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue::646 647 $ cat /sys/devices/system/cpu/cpu4/hotplug/state648 169649 $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target650 $ cat /sys/devices/system/cpu/cpu4/hotplug/state651 140652 653It is important to note that the teardown callback of state 140 have been654invoked. And now get back online::655 656 $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target657 $ cat /sys/devices/system/cpu/cpu4/hotplug/state658 169659 660With trace events enabled, the individual steps are visible, too::661 662 # TASK-PID CPU# TIMESTAMP FUNCTION663 # | | | | |664 bash-394 [001] 22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)665 cpuhp/4-31 [004] 22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)666 cpuhp/4-31 [004] 22.990: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0667 cpuhp/4-31 [004] 22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)668 cpuhp/4-31 [004] 22.992: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0669 cpuhp/4-31 [004] 22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)670 cpuhp/4-31 [004] 22.994: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0671 cpuhp/4-31 [004] 22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)672 cpuhp/4-31 [004] 22.996: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0673 bash-394 [001] 22.997: cpuhp_exit: cpu: 0004 state: 140 step: 169 ret: 0674 bash-394 [005] 95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)675 cpuhp/4-31 [004] 95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)676 cpuhp/4-31 [004] 95.542: cpuhp_exit: cpu: 0004 state: 141 step: 141 ret: 0677 cpuhp/4-31 [004] 95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)678 cpuhp/4-31 [004] 95.544: cpuhp_exit: cpu: 0004 state: 142 step: 142 ret: 0679 cpuhp/4-31 [004] 95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)680 cpuhp/4-31 [004] 95.546: cpuhp_exit: cpu: 0004 state: 143 step: 143 ret: 0681 cpuhp/4-31 [004] 95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)682 cpuhp/4-31 [004] 95.548: cpuhp_exit: cpu: 0004 state: 144 step: 144 ret: 0683 cpuhp/4-31 [004] 95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)684 cpuhp/4-31 [004] 95.550: cpuhp_exit: cpu: 0004 state: 145 step: 145 ret: 0685 cpuhp/4-31 [004] 95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)686 cpuhp/4-31 [004] 95.552: cpuhp_exit: cpu: 0004 state: 168 step: 168 ret: 0687 bash-394 [005] 95.553: cpuhp_exit: cpu: 0004 state: 169 step: 140 ret: 0688 689As it an be seen, CPU4 went down until timestamp 22.996 and then back up until69095.552. All invoked callbacks including their return codes are visible in the691trace.692 693Architecture's requirements694===========================695 696The following functions and configurations are required:697 698``CONFIG_HOTPLUG_CPU``699 This entry needs to be enabled in Kconfig700 701``__cpu_up()``702 Arch interface to bring up a CPU703 704``__cpu_disable()``705 Arch interface to shutdown a CPU, no more interrupts can be handled by the706 kernel after the routine returns. This includes the shutdown of the timer.707 708``__cpu_die()``709 This actually supposed to ensure death of the CPU. Actually look at some710 example code in other arch that implement CPU hotplug. The processor is taken711 down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``712 typically waits for some per_cpu state to be set, to ensure the processor dead713 routine is called to be sure positively.714 715User Space Notification716=======================717 718After CPU successfully onlined or offline udev events are sent. A udev rule like::719 720 SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"721 722will receive all events. A script like::723 724 #!/bin/sh725 726 if [ "${ACTION}" = "offline" ]727 then728 echo "CPU ${DEVPATH##*/} offline"729 730 elif [ "${ACTION}" = "online" ]731 then732 echo "CPU ${DEVPATH##*/} online"733 734 fi735 736can process the event further.737 738When changes to the CPUs in the system occur, the sysfs file739/sys/devices/system/cpu/crash_hotplug contains '1' if the kernel740updates the kdump capture kernel list of CPUs itself (via elfcorehdr and741other relevant kexec segment), or '0' if userspace must update the kdump742capture kernel list of CPUs.743 744The availability depends on the CONFIG_HOTPLUG_CPU kernel configuration745option.746 747To skip userspace processing of CPU hot un/plug events for kdump748(i.e. the unload-then-reload to obtain a current list of CPUs), this sysfs749file can be used in a udev rule as follows:750 751 SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"752 753For a CPU hot un/plug event, if the architecture supports kernel updates754of the elfcorehdr (which contains the list of CPUs) and other relevant755kexec segments, then the rule skips the unload-then-reload of the kdump756capture kernel.757 758Kernel Inline Documentations Reference759======================================760 761.. kernel-doc:: include/linux/cpuhotplug.h762