257 lines · plain
1=================2Freezing of tasks3=================4 5(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL6 7I. What is the freezing of tasks?8=================================9 10The freezing of tasks is a mechanism by which user space processes and some11kernel threads are controlled during hibernation or system-wide suspend (on some12architectures).13 14II. How does it work?15=====================16 17There is one per-task flag (PF_NOFREEZE) and three per-task states18(TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE) used for that.19The tasks that have PF_NOFREEZE unset (all user space tasks and some kernel20threads) are regarded as 'freezable' and treated in a special way before the21system enters a sleep state as well as before a hibernation image is created22(hibernation is directly covered by what follows, but the description applies23to system-wide suspend too).24 25Namely, as the first step of the hibernation procedure the function26freeze_processes() (defined in kernel/power/process.c) is called. A system-wide27static key freezer_active (as opposed to a per-task flag or state) is used to28indicate whether the system is to undergo a freezing operation. And29freeze_processes() sets this static key. After this, it executes30try_to_freeze_tasks() that sends a fake signal to all user space processes, and31wakes up all the kernel threads. All freezable tasks must react to that by32calling try_to_freeze(), which results in a call to __refrigerator() (defined33in kernel/freezer.c), which changes the task's state to TASK_FROZEN, and makes34it loop until it is woken by an explicit TASK_FROZEN wakeup. Then, that task35is regarded as 'frozen' and so the set of functions handling this mechanism is36referred to as 'the freezer' (these functions are defined in37kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). User space38tasks are generally frozen before kernel threads.39 40__refrigerator() must not be called directly. Instead, use the41try_to_freeze() function (defined in include/linux/freezer.h), that checks42if the task is to be frozen and makes the task enter __refrigerator().43 44For user space processes try_to_freeze() is called automatically from the45signal-handling code, but the freezable kernel threads need to call it46explicitly in suitable places or use the wait_event_freezable() or47wait_event_freezable_timeout() macros (defined in include/linux/wait.h)48that put the task to sleep (TASK_INTERRUPTIBLE) or freeze it (TASK_FROZEN) if49freezer_active is set. The main loop of a freezable kernel thread may look50like the following one::51 52 set_freezable();53 54 while (true) {55 struct task_struct *tsk = NULL;56 57 wait_event_freezable(oom_reaper_wait, oom_reaper_list != NULL);58 spin_lock_irq(&oom_reaper_lock);59 if (oom_reaper_list != NULL) {60 tsk = oom_reaper_list;61 oom_reaper_list = tsk->oom_reaper_list;62 }63 spin_unlock_irq(&oom_reaper_lock);64 65 if (tsk)66 oom_reap_task(tsk);67 }68 69(from mm/oom_kill.c::oom_reaper()).70 71If a freezable kernel thread is not put to the frozen state after the freezer72has initiated a freezing operation, the freezing of tasks will fail and the73entire system-wide transition will be cancelled. For this reason, freezable74kernel threads must call try_to_freeze() somewhere or use one of the75wait_event_freezable() and wait_event_freezable_timeout() macros.76 77After the system memory state has been restored from a hibernation image and78devices have been reinitialized, the function thaw_processes() is called in79order to wake up each frozen task. Then, the tasks that have been frozen leave80__refrigerator() and continue running.81 82 83Rationale behind the functions dealing with freezing and thawing of tasks84-------------------------------------------------------------------------85 86freeze_processes():87 - freezes only userspace tasks88 89freeze_kernel_threads():90 - freezes all tasks (including kernel threads) because we can't freeze91 kernel threads without freezing userspace tasks92 93thaw_kernel_threads():94 - thaws only kernel threads; this is particularly useful if we need to do95 anything special in between thawing of kernel threads and thawing of96 userspace tasks, or if we want to postpone the thawing of userspace tasks97 98thaw_processes():99 - thaws all tasks (including kernel threads) because we can't thaw userspace100 tasks without thawing kernel threads101 102 103III. Which kernel threads are freezable?104========================================105 106Kernel threads are not freezable by default. However, a kernel thread may clear107PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE108directly is not allowed). From this point it is regarded as freezable109and must call try_to_freeze() or variants of wait_event_freezable() in a110suitable place.111 112IV. Why do we do that?113======================114 115Generally speaking, there is a couple of reasons to use the freezing of tasks:116 1171. The principal reason is to prevent filesystems from being damaged after118 hibernation. At the moment we have no simple means of checkpointing119 filesystems, so if there are any modifications made to filesystem data and/or120 metadata on disks, we cannot bring them back to the state from before the121 modifications. At the same time each hibernation image contains some122 filesystem-related information that must be consistent with the state of the123 on-disk data and metadata after the system memory state has been restored124 from the image (otherwise the filesystems will be damaged in a nasty way,125 usually making them almost impossible to repair). We therefore freeze126 tasks that might cause the on-disk filesystems' data and metadata to be127 modified after the hibernation image has been created and before the128 system is finally powered off. The majority of these are user space129 processes, but if any of the kernel threads may cause something like this130 to happen, they have to be freezable.131 1322. Next, to create the hibernation image we need to free a sufficient amount of133 memory (approximately 50% of available RAM) and we need to do that before134 devices are deactivated, because we generally need them for swapping out.135 Then, after the memory for the image has been freed, we don't want tasks136 to allocate additional memory and we prevent them from doing that by137 freezing them earlier. [Of course, this also means that device drivers138 should not allocate substantial amounts of memory from their .suspend()139 callbacks before hibernation, but this is a separate issue.]140 1413. The third reason is to prevent user space processes and some kernel threads142 from interfering with the suspending and resuming of devices. A user space143 process running on a second CPU while we are suspending devices may, for144 example, be troublesome and without the freezing of tasks we would need some145 safeguards against race conditions that might occur in such a case.146 147Although Linus Torvalds doesn't like the freezing of tasks, he said this in one148of the discussions on LKML (https://lore.kernel.org/r/alpine.LFD.0.98.0704271801020.9964@woody.linux-foundation.org):149 150"RJW:> Why we freeze tasks at all or why we freeze kernel threads?151 152Linus: In many ways, 'at all'.153 154I **do** realize the IO request queue issues, and that we cannot actually do155s2ram with some devices in the middle of a DMA. So we want to be able to156avoid *that*, there's no question about that. And I suspect that stopping157user threads and then waiting for a sync is practically one of the easier158ways to do so.159 160So in practice, the 'at all' may become a 'why freeze kernel threads?' and161freezing user threads I don't find really objectionable."162 163Still, there are kernel threads that may want to be freezable. For example, if164a kernel thread that belongs to a device driver accesses the device directly, it165in principle needs to know when the device is suspended, so that it doesn't try166to access it at that time. However, if the kernel thread is freezable, it will167be frozen before the driver's .suspend() callback is executed and it will be168thawed after the driver's .resume() callback has run, so it won't be accessing169the device while it's suspended.170 1714. Another reason for freezing tasks is to prevent user space processes from172 realizing that hibernation (or suspend) operation takes place. Ideally, user173 space processes should not notice that such a system-wide operation has174 occurred and should continue running without any problems after the restore175 (or resume from suspend). Unfortunately, in the most general case this176 is quite difficult to achieve without the freezing of tasks. Consider,177 for example, a process that depends on all CPUs being online while it's178 running. Since we need to disable nonboot CPUs during the hibernation,179 if this process is not frozen, it may notice that the number of CPUs has180 changed and may start to work incorrectly because of that.181 182V. Are there any problems related to the freezing of tasks?183===========================================================184 185Yes, there are.186 187First of all, the freezing of kernel threads may be tricky if they depend one188on another. For example, if kernel thread A waits for a completion (in the189TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B190and B is frozen in the meantime, then A will be blocked until B is thawed, which191may be undesirable. That's why kernel threads are not freezable by default.192 193Second, there are the following two problems related to the freezing of user194space processes:195 1961. Putting processes into an uninterruptible sleep distorts the load average.1972. Now that we have FUSE, plus the framework for doing device drivers in198 userspace, it gets even more complicated because some userspace processes are199 now doing the sorts of things that kernel threads do200 (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).201 202The problem 1. seems to be fixable, although it hasn't been fixed so far. The203other one is more serious, but it seems that we can work around it by using204hibernation (and suspend) notifiers (in that case, though, we won't be able to205avoid the realization by the user space processes that the hibernation is taking206place).207 208There are also problems that the freezing of tasks tends to expose, although209they are not directly related to it. For example, if request_firmware() is210called from a device driver's .resume() routine, it will timeout and eventually211fail, because the user land process that should respond to the request is frozen212at this point. So, seemingly, the failure is due to the freezing of tasks.213Suppose, however, that the firmware file is located on a filesystem accessible214only through another device that hasn't been resumed yet. In that case,215request_firmware() will fail regardless of whether or not the freezing of tasks216is used. Consequently, the problem is not really related to the freezing of217tasks, since it generally exists anyway.218 219A driver must have all firmwares it may need in RAM before suspend() is called.220If keeping them is not practical, for example due to their size, they must be221requested early enough using the suspend notifier API described in222Documentation/driver-api/pm/notifiers.rst.223 224VI. Are there any precautions to be taken to prevent freezing failures?225=======================================================================226 227Yes, there are.228 229First of all, grabbing the 'system_transition_mutex' lock to mutually exclude a230piece of code from system-wide sleep such as suspend/hibernation is not231encouraged. If possible, that piece of code must instead hook onto the232suspend/hibernation notifiers to achieve mutual exclusion. Look at the233CPU-Hotplug code (kernel/cpu.c) for an example.234 235However, if that is not feasible, and grabbing 'system_transition_mutex' is236deemed necessary, it is strongly discouraged to directly call237mutex_[un]lock(&system_transition_mutex) since that could lead to freezing238failures, because if the suspend/hibernate code successfully acquired the239'system_transition_mutex' lock, and hence that other entity failed to acquire240the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE state. As a241consequence, the freezer would not be able to freeze that task, leading to242freezing failure.243 244However, the [un]lock_system_sleep() APIs are safe to use in this scenario,245since they ask the freezer to skip freezing this task, since it is anyway246"frozen enough" as it is blocked on 'system_transition_mutex', which will be247released only after the entire suspend/hibernation sequence is complete. So, to248summarize, use [un]lock_system_sleep() instead of directly using249mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures.250 251V. Miscellaneous252================253 254/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze255all user space processes or all freezable kernel threads, in unit of256millisecond. The default value is 20000, with range of unsigned integer.257