779 lines · plain
1=========2Workqueue3=========4 5:Date: September, 20106:Author: Tejun Heo <tj@kernel.org>7:Author: Florian Mickler <florian@mickler.org>8 9 10Introduction11============12 13There are many cases where an asynchronous process execution context14is needed and the workqueue (wq) API is the most commonly used15mechanism for such cases.16 17When such an asynchronous execution context is needed, a work item18describing which function to execute is put on a queue. An19independent thread serves as the asynchronous execution context. The20queue is called workqueue and the thread is called worker.21 22While there are work items on the workqueue the worker executes the23functions associated with the work items one after the other. When24there is no work item left on the workqueue the worker becomes idle.25When a new work item gets queued, the worker begins executing again.26 27 28Why Concurrency Managed Workqueue?29==================================30 31In the original wq implementation, a multi threaded (MT) wq had one32worker thread per CPU and a single threaded (ST) wq had one worker33thread system-wide. A single MT wq needed to keep around the same34number of workers as the number of CPUs. The kernel grew a lot of MT35wq users over the years and with the number of CPU cores continuously36rising, some systems saturated the default 32k PID space just booting37up.38 39Although MT wq wasted a lot of resource, the level of concurrency40provided was unsatisfactory. The limitation was common to both ST and41MT wq albeit less severe on MT. Each wq maintained its own separate42worker pool. An MT wq could provide only one execution context per CPU43while an ST wq one for the whole system. Work items had to compete for44those very limited execution contexts leading to various problems45including proneness to deadlocks around the single execution context.46 47The tension between the provided level of concurrency and resource48usage also forced its users to make unnecessary tradeoffs like libata49choosing to use ST wq for polling PIOs and accepting an unnecessary50limitation that no two polling PIOs can progress at the same time. As51MT wq don't provide much better concurrency, users which require52higher level of concurrency, like async or fscache, had to implement53their own thread pool.54 55Concurrency Managed Workqueue (cmwq) is a reimplementation of wq with56focus on the following goals.57 58* Maintain compatibility with the original workqueue API.59 60* Use per-CPU unified worker pools shared by all wq to provide61 flexible level of concurrency on demand without wasting a lot of62 resource.63 64* Automatically regulate worker pool and level of concurrency so that65 the API users don't need to worry about such details.66 67 68The Design69==========70 71In order to ease the asynchronous execution of functions a new72abstraction, the work item, is introduced.73 74A work item is a simple struct that holds a pointer to the function75that is to be executed asynchronously. Whenever a driver or subsystem76wants a function to be executed asynchronously it has to set up a work77item pointing to that function and queue that work item on a78workqueue.79 80A work item can be executed in either a thread or the BH (softirq) context.81 82For threaded workqueues, special purpose threads, called [k]workers, execute83the functions off of the queue, one after the other. If no work is queued,84the worker threads become idle. These worker threads are managed in85worker-pools.86 87The cmwq design differentiates between the user-facing workqueues that88subsystems and drivers queue work items on and the backend mechanism89which manages worker-pools and processes the queued work items.90 91There are two worker-pools, one for normal work items and the other92for high priority ones, for each possible CPU and some extra93worker-pools to serve work items queued on unbound workqueues - the94number of these backing pools is dynamic.95 96BH workqueues use the same framework. However, as there can only be one97concurrent execution context, there's no need to worry about concurrency.98Each per-CPU BH worker pool contains only one pseudo worker which represents99the BH execution context. A BH workqueue can be considered a convenience100interface to softirq.101 102Subsystems and drivers can create and queue work items through special103workqueue API functions as they see fit. They can influence some104aspects of the way the work items are executed by setting flags on the105workqueue they are putting the work item on. These flags include106things like CPU locality, concurrency limits, priority and more. To107get a detailed overview refer to the API description of108``alloc_workqueue()`` below.109 110When a work item is queued to a workqueue, the target worker-pool is111determined according to the queue parameters and workqueue attributes112and appended on the shared worklist of the worker-pool. For example,113unless specifically overridden, a work item of a bound workqueue will114be queued on the worklist of either normal or highpri worker-pool that115is associated to the CPU the issuer is running on.116 117For any thread pool implementation, managing the concurrency level118(how many execution contexts are active) is an important issue. cmwq119tries to keep the concurrency at a minimal but sufficient level.120Minimal to save resources and sufficient in that the system is used at121its full capacity.122 123Each worker-pool bound to an actual CPU implements concurrency124management by hooking into the scheduler. The worker-pool is notified125whenever an active worker wakes up or sleeps and keeps track of the126number of the currently runnable workers. Generally, work items are127not expected to hog a CPU and consume many cycles. That means128maintaining just enough concurrency to prevent work processing from129stalling should be optimal. As long as there are one or more runnable130workers on the CPU, the worker-pool doesn't start execution of a new131work, but, when the last running worker goes to sleep, it immediately132schedules a new worker so that the CPU doesn't sit idle while there133are pending work items. This allows using a minimal number of workers134without losing execution bandwidth.135 136Keeping idle workers around doesn't cost other than the memory space137for kthreads, so cmwq holds onto idle ones for a while before killing138them.139 140For unbound workqueues, the number of backing pools is dynamic.141Unbound workqueue can be assigned custom attributes using142``apply_workqueue_attrs()`` and workqueue will automatically create143backing worker pools matching the attributes. The responsibility of144regulating concurrency level is on the users. There is also a flag to145mark a bound wq to ignore the concurrency management. Please refer to146the API section for details.147 148Forward progress guarantee relies on that workers can be created when149more execution contexts are necessary, which in turn is guaranteed150through the use of rescue workers. All work items which might be used151on code paths that handle memory reclaim are required to be queued on152wq's that have a rescue-worker reserved for execution under memory153pressure. Else it is possible that the worker-pool deadlocks waiting154for execution contexts to free up.155 156 157Application Programming Interface (API)158=======================================159 160``alloc_workqueue()`` allocates a wq. The original161``create_*workqueue()`` functions are deprecated and scheduled for162removal. ``alloc_workqueue()`` takes three arguments - ``@name``,163``@flags`` and ``@max_active``. ``@name`` is the name of the wq and164also used as the name of the rescuer thread if there is one.165 166A wq no longer manages execution resources but serves as a domain for167forward progress guarantee, flush and work item attributes. ``@flags``168and ``@max_active`` control how work items are assigned execution169resources, scheduled and executed.170 171 172``flags``173---------174 175``WQ_BH``176 BH workqueues can be considered a convenience interface to softirq. BH177 workqueues are always per-CPU and all BH work items are executed in the178 queueing CPU's softirq context in the queueing order.179 180 All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the181 only allowed additional flag.182 183 BH work items cannot sleep. All other features such as delayed queueing,184 flushing and canceling are supported.185 186``WQ_UNBOUND``187 Work items queued to an unbound wq are served by the special188 worker-pools which host workers which are not bound to any189 specific CPU. This makes the wq behave as a simple execution190 context provider without concurrency management. The unbound191 worker-pools try to start execution of work items as soon as192 possible. Unbound wq sacrifices locality but is useful for193 the following cases.194 195 * Wide fluctuation in the concurrency level requirement is196 expected and using bound wq may end up creating large number197 of mostly unused workers across different CPUs as the issuer198 hops through different CPUs.199 200 * Long running CPU intensive workloads which can be better201 managed by the system scheduler.202 203``WQ_FREEZABLE``204 A freezable wq participates in the freeze phase of the system205 suspend operations. Work items on the wq are drained and no206 new work item starts execution until thawed.207 208``WQ_MEM_RECLAIM``209 All wq which might be used in the memory reclaim paths **MUST**210 have this flag set. The wq is guaranteed to have at least one211 execution context regardless of memory pressure.212 213``WQ_HIGHPRI``214 Work items of a highpri wq are queued to the highpri215 worker-pool of the target cpu. Highpri worker-pools are216 served by worker threads with elevated nice level.217 218 Note that normal and highpri worker-pools don't interact with219 each other. Each maintains its separate pool of workers and220 implements concurrency management among its workers.221 222``WQ_CPU_INTENSIVE``223 Work items of a CPU intensive wq do not contribute to the224 concurrency level. In other words, runnable CPU intensive225 work items will not prevent other work items in the same226 worker-pool from starting execution. This is useful for bound227 work items which are expected to hog CPU cycles so that their228 execution is regulated by the system scheduler.229 230 Although CPU intensive work items don't contribute to the231 concurrency level, start of their executions is still232 regulated by the concurrency management and runnable233 non-CPU-intensive work items can delay execution of CPU234 intensive work items.235 236 This flag is meaningless for unbound wq.237 238 239``max_active``240--------------241 242``@max_active`` determines the maximum number of execution contexts per243CPU which can be assigned to the work items of a wq. For example, with244``@max_active`` of 16, at most 16 work items of the wq can be executing245at the same time per CPU. This is always a per-CPU attribute, even for246unbound workqueues.247 248The maximum limit for ``@max_active`` is 512 and the default value used249when 0 is specified is 256. These values are chosen sufficiently high250such that they are not the limiting factor while providing protection in251runaway cases.252 253The number of active work items of a wq is usually regulated by the254users of the wq, more specifically, by how many work items the users255may queue at the same time. Unless there is a specific need for256throttling the number of active work items, specifying '0' is257recommended.258 259Some users depend on strict execution ordering where only one work item260is in flight at any given time and the work items are processed in261queueing order. While the combination of ``@max_active`` of 1 and262``WQ_UNBOUND`` used to achieve this behavior, this is no longer the263case. Use alloc_ordered_workqueue() instead.264 265 266Example Execution Scenarios267===========================268 269The following example execution scenarios try to illustrate how cmwq270behave under different configurations.271 272 Work items w0, w1, w2 are queued to a bound wq q0 on the same CPU.273 w0 burns CPU for 5ms then sleeps for 10ms then burns CPU for 5ms274 again before finishing. w1 and w2 burn CPU for 5ms then sleep for275 10ms.276 277Ignoring all other tasks, works and processing overhead, and assuming278simple FIFO scheduling, the following is one highly simplified version279of possible sequences of events with the original wq. ::280 281 TIME IN MSECS EVENT282 0 w0 starts and burns CPU283 5 w0 sleeps284 15 w0 wakes up and burns CPU285 20 w0 finishes286 20 w1 starts and burns CPU287 25 w1 sleeps288 35 w1 wakes up and finishes289 35 w2 starts and burns CPU290 40 w2 sleeps291 50 w2 wakes up and finishes292 293And with cmwq with ``@max_active`` >= 3, ::294 295 TIME IN MSECS EVENT296 0 w0 starts and burns CPU297 5 w0 sleeps298 5 w1 starts and burns CPU299 10 w1 sleeps300 10 w2 starts and burns CPU301 15 w2 sleeps302 15 w0 wakes up and burns CPU303 20 w0 finishes304 20 w1 wakes up and finishes305 25 w2 wakes up and finishes306 307If ``@max_active`` == 2, ::308 309 TIME IN MSECS EVENT310 0 w0 starts and burns CPU311 5 w0 sleeps312 5 w1 starts and burns CPU313 10 w1 sleeps314 15 w0 wakes up and burns CPU315 20 w0 finishes316 20 w1 wakes up and finishes317 20 w2 starts and burns CPU318 25 w2 sleeps319 35 w2 wakes up and finishes320 321Now, let's assume w1 and w2 are queued to a different wq q1 which has322``WQ_CPU_INTENSIVE`` set, ::323 324 TIME IN MSECS EVENT325 0 w0 starts and burns CPU326 5 w0 sleeps327 5 w1 and w2 start and burn CPU328 10 w1 sleeps329 15 w2 sleeps330 15 w0 wakes up and burns CPU331 20 w0 finishes332 20 w1 wakes up and finishes333 25 w2 wakes up and finishes334 335 336Guidelines337==========338 339* Do not forget to use ``WQ_MEM_RECLAIM`` if a wq may process work340 items which are used during memory reclaim. Each wq with341 ``WQ_MEM_RECLAIM`` set has an execution context reserved for it. If342 there is dependency among multiple work items used during memory343 reclaim, they should be queued to separate wq each with344 ``WQ_MEM_RECLAIM``.345 346* Unless strict ordering is required, there is no need to use ST wq.347 348* Unless there is a specific need, using 0 for @max_active is349 recommended. In most use cases, concurrency level usually stays350 well under the default limit.351 352* A wq serves as a domain for forward progress guarantee353 (``WQ_MEM_RECLAIM``, flush and work item attributes. Work items354 which are not involved in memory reclaim and don't need to be355 flushed as a part of a group of work items, and don't require any356 special attribute, can use one of the system wq. There is no357 difference in execution characteristics between using a dedicated wq358 and a system wq.359 360* Unless work items are expected to consume a huge amount of CPU361 cycles, using a bound wq is usually beneficial due to the increased362 level of locality in wq operations and work item execution.363 364 365Affinity Scopes366===============367 368An unbound workqueue groups CPUs according to its affinity scope to improve369cache locality. For example, if a workqueue is using the default affinity370scope of "cache", it will group CPUs according to last level cache371boundaries. A work item queued on the workqueue will be assigned to a worker372on one of the CPUs which share the last level cache with the issuing CPU.373Once started, the worker may or may not be allowed to move outside the scope374depending on the ``affinity_strict`` setting of the scope.375 376Workqueue currently supports the following affinity scopes.377 378``default``379 Use the scope in module parameter ``workqueue.default_affinity_scope``380 which is always set to one of the scopes below.381 382``cpu``383 CPUs are not grouped. A work item issued on one CPU is processed by a384 worker on the same CPU. This makes unbound workqueues behave as per-cpu385 workqueues without concurrency management.386 387``smt``388 CPUs are grouped according to SMT boundaries. This usually means that the389 logical threads of each physical CPU core are grouped together.390 391``cache``392 CPUs are grouped according to cache boundaries. Which specific cache393 boundary is used is determined by the arch code. L3 is used in a lot of394 cases. This is the default affinity scope.395 396``numa``397 CPUs are grouped according to NUMA boundaries.398 399``system``400 All CPUs are put in the same group. Workqueue makes no effort to process a401 work item on a CPU close to the issuing CPU.402 403The default affinity scope can be changed with the module parameter404``workqueue.default_affinity_scope`` and a specific workqueue's affinity405scope can be changed using ``apply_workqueue_attrs()``.406 407If ``WQ_SYSFS`` is set, the workqueue will have the following affinity scope408related interface files under its ``/sys/devices/virtual/workqueue/WQ_NAME/``409directory.410 411``affinity_scope``412 Read to see the current affinity scope. Write to change.413 414 When default is the current scope, reading this file will also show the415 current effective scope in parentheses, for example, ``default (cache)``.416 417``affinity_strict``418 0 by default indicating that affinity scopes are not strict. When a work419 item starts execution, workqueue makes a best-effort attempt to ensure420 that the worker is inside its affinity scope, which is called421 repatriation. Once started, the scheduler is free to move the worker422 anywhere in the system as it sees fit. This enables benefiting from scope423 locality while still being able to utilize other CPUs if necessary and424 available.425 426 If set to 1, all workers of the scope are guaranteed always to be in the427 scope. This may be useful when crossing affinity scopes has other428 implications, for example, in terms of power consumption or workload429 isolation. Strict NUMA scope can also be used to match the workqueue430 behavior of older kernels.431 432 433Affinity Scopes and Performance434===============================435 436It'd be ideal if an unbound workqueue's behavior is optimal for vast437majority of use cases without further tuning. Unfortunately, in the current438kernel, there exists a pronounced trade-off between locality and utilization439necessitating explicit configurations when workqueues are heavily used.440 441Higher locality leads to higher efficiency where more work is performed for442the same number of consumed CPU cycles. However, higher locality may also443cause lower overall system utilization if the work items are not spread444enough across the affinity scopes by the issuers. The following performance445testing with dm-crypt clearly illustrates this trade-off.446 447The tests are run on a CPU with 12-cores/24-threads split across four L3448caches (AMD Ryzen 9 3900x). CPU clock boost is turned off for consistency.449``/dev/dm-0`` is a dm-crypt device created on NVME SSD (Samsung 990 PRO) and450opened with ``cryptsetup`` with default settings.451 452 453Scenario 1: Enough issuers and work spread across the machine454-------------------------------------------------------------455 456The command used: ::457 458 $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k --ioengine=libaio \459 --iodepth=64 --runtime=60 --numjobs=24 --time_based --group_reporting \460 --name=iops-test-job --verify=sha512461 462There are 24 issuers, each issuing 64 IOs concurrently. ``--verify=sha512``463makes ``fio`` generate and read back the content each time which makes464execution locality matter between the issuer and ``kcryptd``. The following465are the read bandwidths and CPU utilizations depending on different affinity466scope settings on ``kcryptd`` measured over five runs. Bandwidths are in467MiBps, and CPU util in percents.468 469.. list-table::470 :widths: 16 20 20471 :header-rows: 1472 473 * - Affinity474 - Bandwidth (MiBps)475 - CPU util (%)476 477 * - system478 - 1159.40 ±1.34479 - 99.31 ±0.02480 481 * - cache482 - 1166.40 ±0.89483 - 99.34 ±0.01484 485 * - cache (strict)486 - 1166.00 ±0.71487 - 99.35 ±0.01488 489With enough issuers spread across the system, there is no downside to490"cache", strict or otherwise. All three configurations saturate the whole491machine but the cache-affine ones outperform by 0.6% thanks to improved492locality.493 494 495Scenario 2: Fewer issuers, enough work for saturation496-----------------------------------------------------497 498The command used: ::499 500 $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k \501 --ioengine=libaio --iodepth=64 --runtime=60 --numjobs=8 \502 --time_based --group_reporting --name=iops-test-job --verify=sha512503 504The only difference from the previous scenario is ``--numjobs=8``. There are505a third of the issuers but is still enough total work to saturate the506system.507 508.. list-table::509 :widths: 16 20 20510 :header-rows: 1511 512 * - Affinity513 - Bandwidth (MiBps)514 - CPU util (%)515 516 * - system517 - 1155.40 ±0.89518 - 97.41 ±0.05519 520 * - cache521 - 1154.40 ±1.14522 - 96.15 ±0.09523 524 * - cache (strict)525 - 1112.00 ±4.64526 - 93.26 ±0.35527 528This is more than enough work to saturate the system. Both "system" and529"cache" are nearly saturating the machine but not fully. "cache" is using530less CPU but the better efficiency puts it at the same bandwidth as531"system".532 533Eight issuers moving around over four L3 cache scope still allow "cache534(strict)" to mostly saturate the machine but the loss of work conservation535is now starting to hurt with 3.7% bandwidth loss.536 537 538Scenario 3: Even fewer issuers, not enough work to saturate539-----------------------------------------------------------540 541The command used: ::542 543 $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k \544 --ioengine=libaio --iodepth=64 --runtime=60 --numjobs=4 \545 --time_based --group_reporting --name=iops-test-job --verify=sha512546 547Again, the only difference is ``--numjobs=4``. With the number of issuers548reduced to four, there now isn't enough work to saturate the whole system549and the bandwidth becomes dependent on completion latencies.550 551.. list-table::552 :widths: 16 20 20553 :header-rows: 1554 555 * - Affinity556 - Bandwidth (MiBps)557 - CPU util (%)558 559 * - system560 - 993.60 ±1.82561 - 75.49 ±0.06562 563 * - cache564 - 973.40 ±1.52565 - 74.90 ±0.07566 567 * - cache (strict)568 - 828.20 ±4.49569 - 66.84 ±0.29570 571Now, the tradeoff between locality and utilization is clearer. "cache" shows5722% bandwidth loss compared to "system" and "cache (struct)" whopping 20%.573 574 575Conclusion and Recommendations576------------------------------577 578In the above experiments, the efficiency advantage of the "cache" affinity579scope over "system" is, while consistent and noticeable, small. However, the580impact is dependent on the distances between the scopes and may be more581pronounced in processors with more complex topologies.582 583While the loss of work-conservation in certain scenarios hurts, it is a lot584better than "cache (strict)" and maximizing workqueue utilization is585unlikely to be the common case anyway. As such, "cache" is the default586affinity scope for unbound pools.587 588* As there is no one option which is great for most cases, workqueue usages589 that may consume a significant amount of CPU are recommended to configure590 the workqueues using ``apply_workqueue_attrs()`` and/or enable591 ``WQ_SYSFS``.592 593* An unbound workqueue with strict "cpu" affinity scope behaves the same as594 ``WQ_CPU_INTENSIVE`` per-cpu workqueue. There is no real advanage to the595 latter and an unbound workqueue provides a lot more flexibility.596 597* Affinity scopes are introduced in Linux v6.5. To emulate the previous598 behavior, use strict "numa" affinity scope.599 600* The loss of work-conservation in non-strict affinity scopes is likely601 originating from the scheduler. There is no theoretical reason why the602 kernel wouldn't be able to do the right thing and maintain603 work-conservation in most cases. As such, it is possible that future604 scheduler improvements may make most of these tunables unnecessary.605 606 607Examining Configuration608=======================609 610Use tools/workqueue/wq_dump.py to examine unbound CPU affinity611configuration, worker pools and how workqueues map to the pools: ::612 613 $ tools/workqueue/wq_dump.py614 Affinity Scopes615 ===============616 wq_unbound_cpumask=0000000f617 618 CPU619 nr_pods 4620 pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008621 pod_node [0]=0 [1]=0 [2]=1 [3]=1622 cpu_pod [0]=0 [1]=1 [2]=2 [3]=3623 624 SMT625 nr_pods 4626 pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008627 pod_node [0]=0 [1]=0 [2]=1 [3]=1628 cpu_pod [0]=0 [1]=1 [2]=2 [3]=3629 630 CACHE (default)631 nr_pods 2632 pod_cpus [0]=00000003 [1]=0000000c633 pod_node [0]=0 [1]=1634 cpu_pod [0]=0 [1]=0 [2]=1 [3]=1635 636 NUMA637 nr_pods 2638 pod_cpus [0]=00000003 [1]=0000000c639 pod_node [0]=0 [1]=1640 cpu_pod [0]=0 [1]=0 [2]=1 [3]=1641 642 SYSTEM643 nr_pods 1644 pod_cpus [0]=0000000f645 pod_node [0]=-1646 cpu_pod [0]=0 [1]=0 [2]=0 [3]=0647 648 Worker Pools649 ============650 pool[00] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 0651 pool[01] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 0652 pool[02] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 1653 pool[03] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 1654 pool[04] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 2655 pool[05] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 2656 pool[06] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 3657 pool[07] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 3658 pool[08] ref=42 nice= 0 idle/workers= 6/ 6 cpus=0000000f659 pool[09] ref=28 nice= 0 idle/workers= 3/ 3 cpus=00000003660 pool[10] ref=28 nice= 0 idle/workers= 17/ 17 cpus=0000000c661 pool[11] ref= 1 nice=-20 idle/workers= 1/ 1 cpus=0000000f662 pool[12] ref= 2 nice=-20 idle/workers= 1/ 1 cpus=00000003663 pool[13] ref= 2 nice=-20 idle/workers= 1/ 1 cpus=0000000c664 665 Workqueue CPU -> pool666 =====================667 [ workqueue \ CPU 0 1 2 3 dfl]668 events percpu 0 2 4 6669 events_highpri percpu 1 3 5 7670 events_long percpu 0 2 4 6671 events_unbound unbound 9 9 10 10 8672 events_freezable percpu 0 2 4 6673 events_power_efficient percpu 0 2 4 6674 events_freezable_pwr_ef percpu 0 2 4 6675 rcu_gp percpu 0 2 4 6676 rcu_par_gp percpu 0 2 4 6677 slub_flushwq percpu 0 2 4 6678 netns ordered 8 8 8 8 8679 ...680 681See the command's help message for more info.682 683 684Monitoring685==========686 687Use tools/workqueue/wq_monitor.py to monitor workqueue operations: ::688 689 $ tools/workqueue/wq_monitor.py events690 total infl CPUtime CPUhog CMW/RPR mayday rescued691 events 18545 0 6.1 0 5 - -692 events_highpri 8 0 0.0 0 0 - -693 events_long 3 0 0.0 0 0 - -694 events_unbound 38306 0 0.1 - 7 - -695 events_freezable 0 0 0.0 0 0 - -696 events_power_efficient 29598 0 0.2 0 0 - -697 events_freezable_pwr_ef 10 0 0.0 0 0 - -698 sock_diag_events 0 0 0.0 0 0 - -699 700 total infl CPUtime CPUhog CMW/RPR mayday rescued701 events 18548 0 6.1 0 5 - -702 events_highpri 8 0 0.0 0 0 - -703 events_long 3 0 0.0 0 0 - -704 events_unbound 38322 0 0.1 - 7 - -705 events_freezable 0 0 0.0 0 0 - -706 events_power_efficient 29603 0 0.2 0 0 - -707 events_freezable_pwr_ef 10 0 0.0 0 0 - -708 sock_diag_events 0 0 0.0 0 0 - -709 710 ...711 712See the command's help message for more info.713 714 715Debugging716=========717 718Because the work functions are executed by generic worker threads719there are a few tricks needed to shed some light on misbehaving720workqueue users.721 722Worker threads show up in the process list as: ::723 724 root 5671 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/0:1]725 root 5672 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/1:2]726 root 5673 0.0 0.0 0 0 ? S 12:12 0:00 [kworker/0:0]727 root 5674 0.0 0.0 0 0 ? S 12:13 0:00 [kworker/1:0]728 729If kworkers are going crazy (using too much cpu), there are two types730of possible problems:731 732 1. Something being scheduled in rapid succession733 2. A single work item that consumes lots of cpu cycles734 735The first one can be tracked using tracing: ::736 737 $ echo workqueue:workqueue_queue_work > /sys/kernel/tracing/set_event738 $ cat /sys/kernel/tracing/trace_pipe > out.txt739 (wait a few secs)740 ^C741 742If something is busy looping on work queueing, it would be dominating743the output and the offender can be determined with the work item744function.745 746For the second type of problems it should be possible to just check747the stack trace of the offending worker thread. ::748 749 $ cat /proc/THE_OFFENDING_KWORKER/stack750 751The work item's function should be trivially visible in the stack752trace.753 754 755Non-reentrance Conditions756=========================757 758Workqueue guarantees that a work item cannot be re-entrant if the following759conditions hold after a work item gets queued:760 761 1. The work function hasn't been changed.762 2. No one queues the work item to another workqueue.763 3. The work item hasn't been reinitiated.764 765In other words, if the above conditions hold, the work item is guaranteed to be766executed by at most one worker system-wide at any given time.767 768Note that requeuing the work item (to the same queue) in the self function769doesn't break these conditions, so it's safe to do. Otherwise, caution is770required when breaking the conditions inside a work function.771 772 773Kernel Inline Documentations Reference774======================================775 776.. kernel-doc:: include/linux/workqueue.h777 778.. kernel-doc:: kernel/workqueue.c779