375 lines · plain
1======================2Firmware-Assisted Dump3======================4 5July 20116 7The goal of firmware-assisted dump is to enable the dump of8a crashed system, and to do so from a fully-reset system, and9to minimize the total elapsed time until the system is back10in production use.11 12- Firmware-Assisted Dump (FADump) infrastructure is intended to replace13 the existing phyp assisted dump.14- Fadump uses the same firmware interfaces and memory reservation model15 as phyp assisted dump.16- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore17 in the ELF format in the same way as kdump. This helps us reuse the18 kdump infrastructure for dump capture and filtering.19- Unlike phyp dump, userspace tool does not need to refer any sysfs20 interface while reading /proc/vmcore.21- Unlike phyp dump, FADump allows user to release all the memory reserved22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.23- Once enabled through kernel boot parameter, FADump can be24 started/stopped through /sys/kernel/fadump_registered interface (see25 sysfs files section below) and can be easily integrated with kdump26 service start/stop init scripts.27 28Comparing with kdump or other strategies, firmware-assisted29dump offers several strong, practical advantages:30 31- Unlike kdump, the system has been reset, and loaded32 with a fresh copy of the kernel. In particular,33 PCI and I/O devices have been reinitialized and are34 in a clean, consistent state.35- Once the dump is copied out, the memory that held the dump36 is immediately available to the running kernel. And therefore,37 unlike kdump, FADump doesn't need a 2nd reboot to get back38 the system to the production configuration.39 40The above can only be accomplished by coordination with,41and assistance from the Power firmware. The procedure is42as follows:43 44- The first kernel registers the sections of memory with the45 Power firmware for dump preservation during OS initialization.46 These registered sections of memory are reserved by the first47 kernel during early boot.48 49- When system crashes, the Power firmware will copy the registered50 low memory regions (boot memory) from source to destination area.51 It will also save hardware PTE's.52 53 NOTE:54 The term 'boot memory' means size of the low memory chunk55 that is required for a kernel to boot successfully when56 booted with restricted memory. By default, the boot memory57 size will be the larger of 5% of system RAM or 256MB.58 Alternatively, user can also specify boot memory size59 through boot parameter 'crashkernel=' which will override60 the default calculated size. Use this option if default61 boot memory size is not sufficient for second kernel to62 boot successfully. For syntax of crashkernel= parameter,63 refer to Documentation/admin-guide/kdump/kdump.rst. If any64 offset is provided in crashkernel= parameter, it will be65 ignored as FADump uses a predefined offset to reserve memory66 for boot memory dump preservation in case of a crash.67 68- After the low memory (boot memory) area has been saved, the69 firmware will reset PCI and other hardware state. It will70 *not* clear the RAM. It will then launch the bootloader, as71 normal.72 73- The freshly booted kernel will notice that there is a new node74 (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot75 on OPAL platform) in the device tree, indicating that76 there is crash data available from a previous boot. During77 the early boot OS will reserve rest of the memory above78 boot memory size effectively booting with restricted memory79 size. This will make sure that this kernel (also, referred80 to as second kernel or capture kernel) will not touch any81 of the dump memory area.82 83- User-space tools will read /proc/vmcore to obtain the contents84 of memory, which holds the previous crashed kernel dump in ELF85 format. The userspace tools may copy this info to disk, or86 network, nas, san, iscsi, etc. as desired.87 88- Once the userspace tool is done saving dump, it will echo89 '1' to /sys/kernel/fadump_release_mem to release the reserved90 memory back to general use, except the memory required for91 next firmware-assisted dump registration.92 93 e.g.::94 95 # echo 1 > /sys/kernel/fadump_release_mem96 97Please note that the firmware-assisted dump feature98is only available on POWER6 and above systems on pSeries99(PowerVM) platform and POWER9 and above systems with OP940100or later firmware versions on PowerNV (OPAL) platform.101Note that, OPAL firmware exports ibm,opal/dump node when102FADump is supported on PowerNV platform.103 104On OPAL based machines, system first boots into an intermittent105kernel (referred to as petitboot kernel) before booting into the106capture kernel. This kernel would have minimal kernel and/or107userspace support to process crash data. Such kernel needs to108preserve previously crash'ed kernel's memory for the subsequent109capture kernel boot to process this crash data. Kernel config110option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel111to ensure that crash data is preserved to process later.112 113-- On OPAL based machines (PowerNV), if the kernel is build with114 CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also115 exported as /sys/firmware/opal/mpipl/core file. This procfs file is116 helpful in debugging OPAL crashes with GDB. The kernel memory117 used for exporting this procfs file can be released by echo'ing118 '1' to /sys/firmware/opal/mpipl/release_core node.119 120 e.g.121 # echo 1 > /sys/firmware/opal/mpipl/release_core122 123Implementation details:124-----------------------125 126During boot, a check is made to see if firmware supports127this feature on that particular machine. If it does, then128we check to see if an active dump is waiting for us. If yes129then everything but boot memory size of RAM is reserved during130early boot (See Fig. 2). This area is released once we finish131collecting the dump from user land scripts (e.g. kdump scripts)132that are run. If there is dump data, then the133/sys/kernel/fadump_release_mem file is created, and the reserved134memory is held.135 136If there is no waiting dump data, then only the memory required to137hold CPU state, HPTE region, boot memory dump, and FADump header is138usually reserved at an offset greater than boot memory size (see Fig. 1).139This area is *not* released: this region will be kept permanently140reserved, so that it can act as a receptacle for a copy of the boot141memory content in addition to CPU state and HPTE region, in the case142a crash does occur.143 144Since this reserved memory area is used only after the system crash,145there is no point in blocking this significant chunk of memory from146production kernel. Hence, the implementation uses the Linux kernel's147Contiguous Memory Allocator (CMA) for memory reservation if CMA is148configured for kernel. With CMA reservation this memory will be149available for applications to use it, while kernel is prevented from150using it. With this FADump will still be able to capture all of the151kernel memory and most of the user space memory except the user pages152that were present in CMA region::153 154 o Memory Reservation during first kernel155 156 Low memory Top of memory157 0 boot memory size |<------ Reserved dump area ----->| |158 | | | Permanent Reservation | |159 V V | | V160 +-----------+-----/ /---+---+----+-----------+-------+----+-----+161 | | |///|////| DUMP | HDR |////| |162 +-----------+-----/ /---+---+----+-----------+-------+----+-----+163 | ^ ^ ^ ^ ^164 | | | | | |165 \ CPU HPTE / | |166 -------------------------------- | |167 Boot memory content gets transferred | |168 to reserved area by firmware at the | |169 time of crash. | |170 FADump Header |171 (meta area) |172 |173 |174 Metadata: This area holds a metadata structure whose175 address is registered with f/w and retrieved in the176 second kernel after crash, on platforms that support177 tags (OPAL). Having such structure with info needed178 to process the crashdump eases dump capture process.179 180 Fig. 1181 182 183 o Memory Reservation during second kernel after crash184 185 Low memory Top of memory186 0 boot memory size |187 | |<------------ Crash preserved area ------------>|188 V V |<--- Reserved dump area --->| |189 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+190 | |ELF| | |///|////| DUMP | HDR |/////| |191 +----+---+--+-----/ /---+---+----+-------+-----+-----+-------+192 | | | | | |193 ----- ------------------------------ ---------------194 \ | |195 \ | |196 \ | |197 \ | ----------------------------198 \ | /199 \ | /200 \ | /201 /proc/vmcore202 203 204 +---+205 |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above206 +---+ figures are not always present. For example, OPAL platform207 does not have CPU & HPTE regions while Metadata region is208 not supported on pSeries currently.209 210 +---+211 |ELF| -> elfcorehdr, it is created in second kernel after crash.212 +---+213 214 Note: Memory from 0 to the boot memory size is used by second kernel215 216 Fig. 2217 218 219Currently the dump will be copied from /proc/vmcore to a new file upon220user intervention. The dump data available through /proc/vmcore will be221in ELF format. Hence the existing kdump infrastructure (kdump scripts)222to save the dump works fine with minor modifications. KDump scripts on223major Distro releases have already been modified to work seamlessly (no224user intervention in saving the dump) when FADump is used, instead of225KDump, as dump mechanism.226 227The tools to examine the dump will be same as the ones228used for kdump.229 230How to enable firmware-assisted dump (FADump):231----------------------------------------------232 2331. Set config option CONFIG_FA_DUMP=y and build kernel.2342. Boot into linux kernel with 'fadump=on' kernel cmdline option.235 By default, FADump reserved memory will be initialized as CMA area.236 Alternatively, user can boot linux kernel with 'fadump=nocma' to237 prevent FADump to use CMA.2383. Optionally, user can also set 'crashkernel=' kernel cmdline239 to specify size of the memory to reserve for boot memory dump240 preservation.241 242NOTE:243 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead244 use 'crashkernel=' to specify size of the memory to reserve245 for boot memory dump preservation.246 2. If firmware-assisted dump fails to reserve memory then it247 will fallback to existing kdump mechanism if 'crashkernel='248 option is set at kernel cmdline.249 3. if user wants to capture all of user space memory and ok with250 reserved memory not available to production system, then251 'fadump=nocma' kernel parameter can be used to fallback to252 old behaviour.253 254Sysfs/debugfs files:255--------------------256 257Firmware-assisted dump feature uses sysfs file system to hold258the control files and debugfs file to display memory reserved region.259 260Here is the list of files under kernel sysfs:261 262 /sys/kernel/fadump_enabled263 This is used to display the FADump status.264 265 - 0 = FADump is disabled266 - 1 = FADump is enabled267 268 This interface can be used by kdump init scripts to identify if269 FADump is enabled in the kernel and act accordingly.270 271 /sys/kernel/fadump_registered272 This is used to display the FADump registration status as well273 as to control (start/stop) the FADump registration.274 275 - 0 = FADump is not registered.276 - 1 = FADump is registered and ready to handle system crash.277 278 To register FADump echo 1 > /sys/kernel/fadump_registered and279 echo 0 > /sys/kernel/fadump_registered for un-register and stop the280 FADump. Once the FADump is un-registered, the system crash will not281 be handled and vmcore will not be captured. This interface can be282 easily integrated with kdump service start/stop.283 284 /sys/kernel/fadump/mem_reserved285 286 This is used to display the memory reserved by FADump for saving the287 crash dump.288 289 /sys/kernel/fadump_release_mem290 This file is available only when FADump is active during291 second kernel. This is used to release the reserved memory292 region that are held for saving crash dump. To release the293 reserved memory echo 1 to it::294 295 echo 1 > /sys/kernel/fadump_release_mem296 297 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region298 file will change to reflect the new memory reservations.299 300 The existing userspace tools (kdump infrastructure) can be easily301 enhanced to use this interface to release the memory reserved for302 dump and continue without 2nd reboot.303 304Note: /sys/kernel/fadump_release_opalcore sysfs has moved to305 /sys/firmware/opal/mpipl/release_core306 307 /sys/firmware/opal/mpipl/release_core308 309 This file is available only on OPAL based machines when FADump is310 active during capture kernel. This is used to release the memory311 used by the kernel to export /sys/firmware/opal/mpipl/core file. To312 release this memory, echo '1' to it:313 314 echo 1 > /sys/firmware/opal/mpipl/release_core315 316Note: The following FADump sysfs files are deprecated.317 318+----------------------------------+--------------------------------+319| Deprecated | Alternative |320+----------------------------------+--------------------------------+321| /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled |322+----------------------------------+--------------------------------+323| /sys/kernel/fadump_registered | /sys/kernel/fadump/registered |324+----------------------------------+--------------------------------+325| /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem |326+----------------------------------+--------------------------------+327 328Here is the list of files under powerpc debugfs:329(Assuming debugfs is mounted on /sys/kernel/debug directory.)330 331 /sys/kernel/debug/powerpc/fadump_region332 This file shows the reserved memory regions if FADump is333 enabled otherwise this file is empty. The output format334 is::335 336 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>337 338 and for kernel DUMP region is:339 340 DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes341 342 e.g.343 Contents when FADump is registered during first kernel::344 345 # cat /sys/kernel/debug/powerpc/fadump_region346 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0347 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0348 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0349 350 Contents when FADump is active during second kernel::351 352 # cat /sys/kernel/debug/powerpc/fadump_region353 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020354 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000355 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000356 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000357 358 359NOTE:360 Please refer to Documentation/filesystems/debugfs.rst on361 how to mount the debugfs filesystem.362 363 364TODO:365-----366 - Need to come up with the better approach to find out more367 accurate boot memory size that is required for a kernel to368 boot successfully when booted with restricted memory.369 370Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>371 372This document is based on the original documentation written for phyp373 374assisted dump by Linas Vepstas and Manish Ahuja.375