brintos / linux-shallow public Read only

1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)2.. [see the bottom of this file for redistribution information]3 4=========================================5How to verify bugs and bisect regressions6=========================================7 8This document describes how to check if some Linux kernel problem occurs in code9currently supported by developers -- to then explain how to locate the change10causing the issue, if it is a regression (e.g. did not happen with earlier11versions).12 13The text aims at people running kernels from mainstream Linux distributions on14commodity hardware who want to report a kernel bug to the upstream Linux15developers. Despite this intent, the instructions work just as well for users16who are already familiar with building their own kernels: they help avoid17mistakes occasionally made even by experienced developers.18 19..20   Note: if you see this note, you are reading the text's source file. You21   might want to switch to a rendered version: it makes it a lot easier to22   read and navigate this document -- especially when you want to look something23   up in the reference section, then jump back to where you left off.24..25   Find the latest rendered version of this text here:26   https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html27 28The essence of the process (aka 'TL;DR')29========================================30 31*[If you are new to building or bisecting Linux, ignore this section and head32over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes33the same commands as this section while describing them in brief fashion. The34steps are nevertheless easy to follow and together with accompanying entries35in a reference section mention many alternatives, pitfalls, and additional36aspects, all of which might be essential in your present case.]*37 38**In case you want to check if a bug is present in code currently supported by39developers**, execute just the *preparations* and *segment 1*; while doing so,40consider the newest Linux kernel you regularly use to be the 'working' kernel.41In the following example that's assumed to be 6.0, which is why its sources42will be used to prepare the .config file.43 44**In case you face a regression**, follow the steps at least till the end of45*segment 2*. Then you can submit a preliminary report -- or continue with46*segment 3*, which describes how to perform a bisection needed for a47full-fledged regression report. In the following example 6.0.13 is assumed to be48the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.049will be considered the 'good' release and used to prepare the .config file.50 51* **Preparations**: set up everything to build your own kernels::52 53    # * Remove any software that depends on externally maintained kernel modules54    #   or builds any automatically during bootup.55    # * Ensure Secure Boot permits booting self-compiled Linux kernels.56    # * If you are not already running the 'working' kernel, reboot into it.57    # * Install compilers and everything else needed for building Linux.58    # * Ensure to have 15 Gigabyte free space in your home directory.59    git clone -o mainline --no-checkout \60      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/61    cd ~/linux/62    git remote add -t master stable \63      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git64    git switch --detach v6.065    # * Hint: if you used an existing clone, ensure no stale .config is around.66    make olddefconfig67    # * Ensure the former command picked the .config of the 'working' kernel.68    # * Connect external hardware (USB keys, tokens, ...), start a VM, bring up69    #   VPNs, mount network shares, and briefly try the feature that is broken.70    yes '' | make localmodconfig71    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'72    ./scripts/config -e CONFIG_LOCALVERSION_AUTO73    # * Note, when short on storage space, check the guide for an alternative:74    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \75      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS76    # * Hint: at this point you might want to adjust the build configuration;77    #   you'll have to, if you are running Debian.78    make olddefconfig79    cp .config ~/kernel-config-working80 81* **Segment 1**: build a kernel from the latest mainline codebase.82 83  This among others checks if the problem was fixed already and which developers84  later need to be told about the problem; in case of a regression, this rules85  out a .config change as root of the problem.86 87  a) Checking out latest mainline code::88 89       cd ~/linux/90       git switch --discard-changes --detach mainline/master91 92  b) Build, install, and boot a kernel::93 94       cp ~/kernel-config-working .config95       make olddefconfig96       make -j $(nproc --all)97       # * Make sure there is enough disk space to hold another kernel:98       df -h /boot/ /lib/modules/99       # * Note: on Arch Linux, its derivatives and a few other distributions100       #   the following commands will do nothing at all or only part of the101       #   job. See the step-by-step guide for further details.102       sudo make modules_install103       command -v installkernel && sudo make install104       # * Check how much space your self-built kernel actually needs, which105       #   enables you to make better estimates later:106       du -ch /boot/*$(make -s kernelrelease)* | tail -n 1107       du -sh /lib/modules/$(make -s kernelrelease)/108       # * Hint: the output of the following command will help you pick the109       #   right kernel from the boot menu:110       make -s kernelrelease | tee -a ~/kernels-built111       reboot112       # * Once booted, ensure you are running the kernel you just built by113       #   checking if the output of the next two commands matches:114       tail -n 1 ~/kernels-built115       uname -r116       cat /proc/sys/kernel/tainted117 118  c) Check if the problem occurs with this kernel as well.119 120* **Segment 2**: ensure the 'good' kernel is also a 'working' kernel.121 122  This among others verifies the trimmed .config file actually works well, as123  bisecting with it otherwise would be a waste of time:124 125  a) Start by checking out the sources of the 'good' version::126 127       cd ~/linux/128       git switch --discard-changes --detach v6.0129 130  b) Build, install, and boot a kernel as described earlier in *segment 1,131     section b* -- just feel free to skip the 'du' commands, as you have a rough132     estimate already.133 134  c) Ensure the feature that regressed with the 'broken' kernel actually works135     with this one.136 137* **Segment 3**: perform and validate the bisection.138 139  a) Retrieve the sources for your 'bad' version::140 141       git remote set-branches --add stable linux-6.1.y142       git fetch stable143 144  b) Initialize the bisection::145 146       cd ~/linux/147       git bisect start148       git bisect good v6.0149       git bisect bad v6.1.5150 151  c) Build, install, and boot a kernel as described earlier in *segment 1,152     section b*.153 154     In case building or booting the kernel fails for unrelated reasons, run155     ``git bisect skip``. In all other outcomes, check if the regressed feature156     works with the newly built kernel. If it does, tell Git by executing157     ``git bisect good``; if it does not, run ``git bisect bad`` instead.158 159     All three commands will make Git check out another commit; then re-execute160     this step (e.g. build, install, boot, and test a kernel to then tell Git161     the outcome). Do so again and again until Git shows which commit broke162     things. If you run short of disk space during this process, check the163     section 'Complementary tasks: cleanup during and after the process'164     below.165 166  d) Once your finished the bisection, put a few things away::167 168       cd ~/linux/169       git bisect log > ~/bisect-log170       cp .config ~/bisection-config-culprit171       git bisect reset172 173  e) Try to verify the bisection result::174 175       git switch --discard-changes --detach mainline/master176       git revert --no-edit cafec0cacaca0177       cp ~/kernel-config-working .config178       ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'179 180    This is optional, as some commits are impossible to revert. But if the181    second command worked flawlessly, build, install, and boot one more kernel182    kernel; just this time skip the first command copying the base .config file183    over, as that already has been taken care off.184 185* **Complementary tasks**: cleanup during and after the process.186 187  a) To avoid running out of disk space during a bisection, you might need to188     remove some kernels you built earlier. You most likely want to keep those189     you built during segment 1 and 2 around for a while, but you will most190     likely no longer need kernels tested during the actual bisection191     (Segment 3 c). You can list them in build order using::192 193       ls -ltr /lib/modules/*-local*194 195    To then for example erase a kernel that identifies itself as196    '6.0-rc1-local-gcafec0cacaca0', use this::197 198       sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0199       sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0200       # * Note, on some distributions kernel-install is missing201       #   or does only part of the job.202 203  b) If you performed a bisection and successfully validated the result, feel204     free to remove all kernels built during the actual bisection (Segment 3 c);205     the kernels you built earlier and later you might want to keep around for206     a week or two.207 208* **Optional task**: test a debug patch or a proposed fix later::209 210    git fetch mainline211    git switch --discard-changes --detach mainline/master212    git apply /tmp/foobars-proposed-fix-v1.patch213    cp ~/kernel-config-working .config214    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'215 216  Build, install, and boot a kernel as described in *segment 1, section b* --217  but this time omit the first command copying the build configuration over,218  as that has been taken care of already.219 220.. _introguide_bissbs:221 222Step-by-step guide on how to verify bugs and bisect regressions223===============================================================224 225This guide describes how to set up your own Linux kernels for investigating bugs226or regressions you intend to report. How far you want to follow the instructions227depends on your issue:228 229Execute all steps till the end of *segment 1* to **verify if your kernel problem230is present in code supported by Linux kernel developers**. If it is, you are all231set to report the bug -- unless it did not happen with earlier kernel versions,232as then your want to at least continue with *segment 2* to **check if the issue233qualifies as regression** which receive priority treatment. Depending on the234outcome you then are ready to report a bug or submit a preliminary regression235report; instead of the latter your could also head straight on and follow236*segment 3* to **perform a bisection** for a full-fledged regression report237developers are obliged to act upon.238 239 :ref:`Preparations: set up everything to build your own kernels <introprep_bissbs>`.240 241 :ref:`Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>`.242 243 :ref:`Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>`.244 245 :ref:`Segment 3: perform a bisection and validate the result <introbisect_bissbs>`.246 247 :ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.248 249 :ref:`Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>`.250 251The steps in each segment illustrate the important aspects of the process, while252a comprehensive reference section holds additional details for almost all of the253steps. The reference section sometimes also outlines alternative approaches,254pitfalls, as well as problems that might occur at the particular step -- and how255to get things rolling again.256 257For further details on how to report Linux kernel issues or regressions check258out Documentation/admin-guide/reporting-issues.rst, which works in conjunction259with this document. It among others explains why you need to verify bugs with260the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6),261even if you face a problem with a kernel from a 'stable/longterm' series262(say 6.0.13).263 264For users facing a regression that document also explains why sending a265preliminary report after segment 2 might be wise, as the regression and its266culprit might be known already. For further details on what actually qualifies267as a regression check out Documentation/admin-guide/reporting-regressions.rst.268 269If you run into any problems while following this guide or have ideas how to270improve it, :ref:`please let the kernel developers know <submit_improvements>`.271 272.. _introprep_bissbs:273 274Preparations: set up everything to build your own kernels275---------------------------------------------------------276 277The following steps lay the groundwork for all further tasks.278 279Note: the instructions assume you are building and testing on the same280machine; if you want to compile the kernel on another system, check281:ref:`Build kernels on a different machine <buildhost_bis>` below.282 283.. _backup_bissbs:284 285* Create a fresh backup and put system repair and restore tools at hand, just286  to be prepared for the unlikely case of something going sideways.287 288  [:ref:`details <backup_bisref>`]289 290.. _vanilla_bissbs:291 292* Remove all software that depends on externally developed kernel drivers or293  builds them automatically. That includes but is not limited to DKMS, openZFS,294  VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module).295 296  [:ref:`details <vanilla_bisref>`]297 298.. _secureboot_bissbs:299 300* On platforms with 'Secure Boot' or similar solutions, prepare everything to301  ensure the system will permit your self-compiled kernel to boot. The302  quickest and easiest way to achieve this on commodity x86 systems is to303  disable such techniques in the BIOS setup utility; alternatively, remove304  their restrictions through a process initiated by305  ``mokutil --disable-validation``.306 307  [:ref:`details <secureboot_bisref>`]308 309.. _rangecheck_bissbs:310 311* Determine the kernel versions considered 'good' and 'bad' throughout this312  guide:313 314  * Do you follow this guide to verify if a bug is present in the code the315    primary developers care for? Then consider the version of the newest kernel316    you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2).317 318  * Do you face a regression, e.g. something broke or works worse after319    switching to a newer kernel version? In that case it depends on the version320    range during which the problem appeared:321 322    * Something regressed when updating from a stable/longterm release323      (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a324      stable/longterm version based on one (say 6.1.5)? Then consider the325      mainline release your working kernel is based on to be the 'good'326      version (e.g. 6.0) and the first version to be broken as the 'bad' one327      (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed328      that 6.0 is fine; this hypothesis will be checked in segment 2.329 330    * Something regressed when switching from one mainline version (say 6.0) to331      a later one (like 6.1-rc1) or a stable/longterm release based on it332      (say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'good' and333      the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'.334 335    * Something regressed when updating within a stable/longterm series (say336      from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and 'bad'337      (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series.338 339  *Note, do not confuse 'good' version with 'working' kernel; the latter term340  throughout this guide will refer to the last kernel that has been working341  fine.*342 343  [:ref:`details <rangecheck_bisref>`]344 345.. _bootworking_bissbs:346 347* Boot into the 'working' kernel and briefly use the apparently broken feature.348 349  [:ref:`details <bootworking_bisref>`]350 351.. _diskspace_bissbs:352 353* Ensure to have enough free space for building Linux. 15 Gigabyte in your home354  directory should typically suffice. If you have less available, be sure to pay355  attention to later steps about retrieving the Linux sources and handling of356  debug symbols: both explain approaches reducing the amount of space, which357  should allow you to master these tasks with about 4 Gigabytes free space.358 359  [:ref:`details <diskspace_bisref>`]360 361.. _buildrequires_bissbs:362 363* Install all software required to build a Linux kernel. Often you will need:364  'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',365  'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The366  reference section shows how to quickly install those on various popular Linux367  distributions.368 369  [:ref:`details <buildrequires_bisref>`]370 371.. _sources_bissbs:372 373* Retrieve the mainline Linux sources; then change into the directory holding374  them, as all further commands in this guide are meant to be executed from375  there.376 377  *Note, the following describe how to retrieve the sources using a full378  mainline clone, which downloads about 2,75 GByte as of early 2024. The*379  :ref:`reference section describes two alternatives <sources_bisref>` *:380  one downloads less than 500 MByte, the other works better with unreliable381  internet connections.*382 383  Execute the following command to retrieve a fresh mainline codebase while384  preparing things to add branches for stable/longterm series later::385 386    git clone -o mainline --no-checkout \387      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/388    cd ~/linux/389    git remote add -t master stable \390      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git391 392  [:ref:`details <sources_bisref>`]393 394.. _stablesources_bissbs:395 396* Is one of the versions you earlier established as 'good' or 'bad' a stable or397  longterm release (say 6.1.5)? Then download the code for the series it belongs398  to ('linux-6.1.y' in this example)::399 400    git remote set-branches --add stable linux-6.1.y401    git fetch stable402 403.. _oldconfig_bissbs:404 405* Start preparing a kernel build configuration (the '.config' file).406 407  Before doing so, ensure you are still running the 'working' kernel an earlier408  step told you to boot; if you are unsure, check the current kernelrelease409  identifier using ``uname -r``.410 411  Afterwards check out the source code for the version earlier established as412  'good'. In the following example command this is assumed to be 6.0; note that413  the version number in this and all later Git commands needs to be prefixed414  with a 'v'::415 416    git switch --discard-changes --detach v6.0417 418  Now create a build configuration file::419 420    make olddefconfig421 422  The kernel build scripts then will try to locate the build configuration file423  for the running kernel and then adjust it for the needs of the kernel sources424  you checked out. While doing so, it will print a few lines you need to check.425 426  Look out for a line starting with '# using defaults found in'. It should be427  followed by a path to a file in '/boot/' that contains the release identifier428  of your currently working kernel. If the line instead continues with something429  like 'arch/x86/configs/x86_64_defconfig', then the build infra failed to find430  the .config file for your running kernel -- in which case you have to put one431  there manually, as explained in the reference section.432 433  In case you can not find such a line, look for one containing '# configuration434  written to .config'. If that's the case you have a stale build configuration435  lying around. Unless you intend to use it, delete it; afterwards run436  'make olddefconfig' again and check if it now picked up the right config file437  as base.438 439  [:ref:`details <oldconfig_bisref>`]440 441.. _localmodconfig_bissbs:442 443* Disable any kernel modules apparently superfluous for your setup. This is444  optional, but especially wise for bisections, as it speeds up the build445  process enormously -- at least unless the .config file picked up in the446  previous step was already tailored to your and your hardware needs, in which447  case you should skip this step.448 449  To prepare the trimming, connect external hardware you occasionally use (USB450  keys, tokens, ...), quickly start a VM, and bring up VPNs. And if you rebooted451  since you started that guide, ensure that you tried using the feature causing452  trouble since you started the system. Only then trim your .config::453 454     yes '' | make localmodconfig455 456  There is a catch to this, as the 'apparently' in initial sentence of this step457  and the preparation instructions already hinted at:458 459  The 'localmodconfig' target easily disables kernel modules for features only460  used occasionally -- like modules for external peripherals not yet connected461  since booting, virtualization software not yet utilized, VPN tunnels, and a462  few other things. That's because some tasks rely on kernel modules Linux only463  loads when you execute tasks like the aforementioned ones for the first time.464 465  This drawback of localmodconfig is nothing you should lose sleep over, but466  something to keep in mind: if something is misbehaving with the kernels built467  during this guide, this is most likely the reason. You can reduce or nearly468  eliminate the risk with tricks outlined in the reference section; but when469  building a kernel just for quick testing purposes this is usually not worth470  spending much effort on, as long as it boots and allows to properly test the471  feature that causes trouble.472 473  [:ref:`details <localmodconfig_bisref>`]474 475.. _tagging_bissbs:476 477* Ensure all the kernels you will build are clearly identifiable using a special478  tag and a unique version number::479 480    ./scripts/config --set-str CONFIG_LOCALVERSION '-local'481    ./scripts/config -e CONFIG_LOCALVERSION_AUTO482 483  [:ref:`details <tagging_bisref>`]484 485.. _debugsymbols_bissbs:486 487* Decide how to handle debug symbols.488 489  In the context of this document it is often wise to enable them, as there is a490  decent chance you will need to decode a stack trace from a 'panic', 'Oops',491  'warning', or 'BUG'::492 493    ./scripts/config -d DEBUG_INFO_NONE -e KALLSYMS_ALL -e DEBUG_KERNEL \494      -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS495 496  But if you are extremely short on storage space, you might want to disable497  debug symbols instead::498 499    ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \500      -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE501 502  [:ref:`details <debugsymbols_bisref>`]503 504.. _configmods_bissbs:505 506* Check if you may want or need to adjust some other kernel configuration507  options:508 509  * Are you running Debian? Then you want to avoid known problems by performing510    additional adjustments explained in the reference section.511 512    [:ref:`details <configmods_distros_bisref>`].513 514  * If you want to influence other aspects of the configuration, do so now using515    your preferred tool. Note, to use make targets like 'menuconfig' or516    'nconfig', you will need to install the development files of ncurses; for517    'xconfig' you likewise need the Qt5 or Qt6 headers.518 519    [:ref:`details <configmods_individual_bisref>`].520 521.. _saveconfig_bissbs:522 523* Reprocess the .config after the latest adjustments and store it in a safe524  place::525 526     make olddefconfig527     cp .config ~/kernel-config-working528 529  [:ref:`details <saveconfig_bisref>`]530 531.. _introlatestcheck_bissbs:532 533Segment 1: try to reproduce the problem with the latest codebase534----------------------------------------------------------------535 536The following steps verify if the problem occurs with the code currently537supported by developers. In case you face a regression, it also checks that the538problem is not caused by some .config change, as reporting the issue then would539be a waste of time. [:ref:`details <introlatestcheck_bisref>`]540 541.. _checkoutmaster_bissbs:542 543* Check out the latest Linux codebase.544 545  * Are your 'good' and 'bad' versions from the same stable or longterm series?546    Then check the `front page of kernel.org <https://kernel.org/>`_: if it547    lists a release from that series without an '[EOL]' tag, checkout the series548    latest version ('linux-6.1.y' in the following example)::549 550      cd ~/linux/551      git switch --discard-changes --detach stable/linux-6.1.y552 553    Your series is unsupported, if is not listed or carrying a 'end of life'554    tag. In that case you might want to check if a successor series (say555    linux-6.2.y) or mainline (see next point) fix the bug.556 557  * In all other cases, run::558 559      cd ~/linux/560      git switch --discard-changes --detach mainline/master561 562  [:ref:`details <checkoutmaster_bisref>`]563 564.. _build_bissbs:565 566* Build the image and the modules of your first kernel using the config file you567  prepared::568 569    cp ~/kernel-config-working .config570    make olddefconfig571    make -j $(nproc --all)572 573  If you want your kernel packaged up as deb, rpm, or tar file, see the574  reference section for alternatives, which obviously will require other575  steps to install as well.576 577  [:ref:`details <build_bisref>`]578 579.. _install_bissbs:580 581* Install your newly built kernel.582 583  Before doing so, consider checking if there is still enough space for it::584 585    df -h /boot/ /lib/modules/586 587  For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how588  much your kernels actually require will be determined later during this guide.589 590  Now install the kernel's modules and its image, which will be stored in591  parallel to the your Linux distribution's kernels::592 593    sudo make modules_install594    command -v installkernel && sudo make install595 596  The second command ideally will take care of three steps required at this597  point: copying the kernel's image to /boot/, generating an initramfs, and598  adding an entry for both to the boot loader's configuration.599 600  Sadly some distributions (among them Arch Linux, its derivatives, and many601  immutable Linux distributions) will perform none or only some of those tasks.602  You therefore want to check if all of them were taken care of and manually603  perform those that were not. The reference section provides further details on604  that; your distribution's documentation might help, too.605 606  Once you figured out the steps needed at this point, consider writing them607  down: if you will build more kernels as described in segment 2 and 3, you will608  have to perform those again after executing ``command -v installkernel [...]``.609 610  [:ref:`details <install_bisref>`]611 612.. _storagespace_bissbs:613 614* In case you plan to follow this guide further, check how much storage space615  the kernel, its modules, and other related files like the initramfs consume::616 617    du -ch /boot/*$(make -s kernelrelease)* | tail -n 1618    du -sh /lib/modules/$(make -s kernelrelease)/619 620  Write down or remember those two values for later: they enable you to prevent621  running out of disk space accidentally during a bisection.622 623  [:ref:`details <storagespace_bisref>`]624 625.. _kernelrelease_bissbs:626 627* Show and store the kernelrelease identifier of the kernel you just built::628 629    make -s kernelrelease | tee -a ~/kernels-built630 631  Remember the identifier momentarily, as it will help you pick the right kernel632  from the boot menu upon restarting.633 634* Reboot into your newly built kernel. To ensure your actually started the one635  you just built, you might want to verify if the output of these commands636  matches::637 638    tail -n 1 ~/kernels-built639    uname -r640 641.. _tainted_bissbs:642 643* Check if the kernel marked itself as 'tainted'::644 645    cat /proc/sys/kernel/tainted646 647  If that command does not return '0', check the reference section, as the cause648  for this might interfere with your testing.649 650  [:ref:`details <tainted_bisref>`]651 652.. _recheckbroken_bissbs:653 654* Verify if your bug occurs with the newly built kernel. If it does not, check655  out the instructions in the reference section to ensure nothing went sideways656  during your tests.657 658  [:ref:`details <recheckbroken_bisref>`]659 660.. _recheckstablebroken_bissbs:661 662* Did you just built a stable or longterm kernel? And were you able to reproduce663  the regression with it? Then you should test the latest mainline codebase as664  well, because the result determines which developers the bug must be submitted665  to.666 667  To prepare that test, check out current mainline::668 669    cd ~/linux/670    git switch --discard-changes --detach mainline/master671 672  Now use the checked out code to build and install another kernel using the673  commands the earlier steps already described in more detail::674 675    cp ~/kernel-config-working .config676    make olddefconfig677    make -j $(nproc --all)678    # * Check if the free space suffices holding another kernel:679    df -h /boot/ /lib/modules/680    sudo make modules_install681    command -v installkernel && sudo make install682    make -s kernelrelease | tee -a ~/kernels-built683    reboot684 685  Confirm you booted the kernel you intended to start and check its tainted686  status::687 688    tail -n 1 ~/kernels-built689    uname -r690    cat /proc/sys/kernel/tainted691 692  Now verify if this kernel is showing the problem. If it does, then you need693  to report the bug to the primary developers; if it does not, report it to the694  stable team. See Documentation/admin-guide/reporting-issues.rst for details.695 696  [:ref:`details <recheckstablebroken_bisref>`]697 698Do you follow this guide to verify if a problem is present in the code699currently supported by Linux kernel developers? Then you are done at this700point. If you later want to remove the kernel you just built, check out701:ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`.702 703In case you face a regression, move on and execute at least the next segment704as well.705 706.. _introworkingcheck_bissbs:707 708Segment 2: check if the kernels you build work fine709---------------------------------------------------710 711In case of a regression, you now want to ensure the trimmed configuration file712you created earlier works as expected; a bisection with the .config file713otherwise would be a waste of time. [:ref:`details <introworkingcheck_bisref>`]714 715.. _recheckworking_bissbs:716 717* Build your own variant of the 'working' kernel and check if the feature that718  regressed works as expected with it.719 720  Start by checking out the sources for the version earlier established as721  'good' (once again assumed to be 6.0 here)::722 723    cd ~/linux/724    git switch --discard-changes --detach v6.0725 726  Now use the checked out code to configure, build, and install another kernel727  using the commands the previous subsection explained in more detail::728 729    cp ~/kernel-config-working .config730    make olddefconfig731    make -j $(nproc --all)732    # * Check if the free space suffices holding another kernel:733    df -h /boot/ /lib/modules/734    sudo make modules_install735    command -v installkernel && sudo make install736    make -s kernelrelease | tee -a ~/kernels-built737    reboot738 739  When the system booted, you may want to verify once again that the740  kernel you started is the one you just built::741 742    tail -n 1 ~/kernels-built743    uname -r744 745  Now check if this kernel works as expected; if not, consult the reference746  section for further instructions.747 748  [:ref:`details <recheckworking_bisref>`]749 750.. _introbisect_bissbs:751 752Segment 3: perform the bisection and validate the result753--------------------------------------------------------754 755With all the preparations and precaution builds taken care of, you are now ready756to begin the bisection. This will make you build quite a few kernels -- usually757about 15 in case you encountered a regression when updating to a newer series758(say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build759configuration created earlier this works a lot faster than many people assume:760overall on average it will often just take about 10 to 15 minutes to compile761each kernel on commodity x86 machines.762 763.. _bisectstart_bissbs:764 765* Start the bisection and tell Git about the versions earlier established as766  'good' (6.0 in the following example command) and 'bad' (6.1.5)::767 768    cd ~/linux/769    git bisect start770    git bisect good v6.0771    git bisect bad v6.1.5772 773  [:ref:`details <bisectstart_bisref>`]774 775.. _bisectbuild_bissbs:776 777* Now use the code Git checked out to build, install, and boot a kernel using778  the commands introduced earlier::779 780    cp ~/kernel-config-working .config781    make olddefconfig782    make -j $(nproc --all)783    # * Check if the free space suffices holding another kernel:784    df -h /boot/ /lib/modules/785    sudo make modules_install786    command -v installkernel && sudo make install787    make -s kernelrelease | tee -a ~/kernels-built788    reboot789 790  If compilation fails for some reason, run ``git bisect skip`` and restart791  executing the stack of commands from the beginning.792 793  In case you skipped the 'test latest codebase' step in the guide, check its794  description as for why the 'df [...]' and 'make -s kernelrelease [...]'795  commands are here.796 797  Important note: the latter command from this point on will print release798  identifiers that might look odd or wrong to you -- which they are not, as it's799  totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0'800  if you bisect between versions 6.1 and 6.2 for example.801 802  [:ref:`details <bisectbuild_bisref>`]803 804.. _bisecttest_bissbs:805 806* Now check if the feature that regressed works in the kernel you just built.807 808  You again might want to start by making sure the kernel you booted is the one809  you just built::810 811    cd ~/linux/812    tail -n 1 ~/kernels-built813    uname -r814 815  Now verify if the feature that regressed works at this kernel bisection point.816  If it does, run this::817 818    git bisect good819 820  If it does not, run this::821 822    git bisect bad823 824  Be sure about what you tell Git, as getting this wrong just once will send the825  rest of the bisection totally off course.826 827  While the bisection is ongoing, Git will use the information you provided to828  find and check out another bisection point for you to test. While doing so, it829  will print something like 'Bisecting: 675 revisions left to test after this830  (roughly 10 steps)' to indicate how many further changes it expects to be831  tested. Now build and install another kernel using the instructions from the832  previous step; afterwards follow the instructions in this step again.833 834  Repeat this again and again until you finish the bisection -- that's the case835  when Git after tagging a change as 'good' or 'bad' prints something like836  'cafecaca0c0dacafecaca0c0dacafecaca0c0da is the first bad commit'; right837  afterwards it will show some details about the culprit including the patch838  description of the change. The latter might fill your terminal screen, so you839  might need to scroll up to see the message mentioning the culprit;840  alternatively, run ``git bisect log > ~/bisection-log``.841 842  [:ref:`details <bisecttest_bisref>`]843 844.. _bisectlog_bissbs:845 846* Store Git's bisection log and the current .config file in a safe place before847  telling Git to reset the sources to the state before the bisection::848 849    cd ~/linux/850    git bisect log > ~/bisection-log851    cp .config ~/bisection-config-culprit852    git bisect reset853 854  [:ref:`details <bisectlog_bisref>`]855 856.. _revert_bissbs:857 858* Try reverting the culprit on top of latest mainline to see if this fixes your859  regression.860 861  This is optional, as it might be impossible or hard to realize. The former is862  the case, if the bisection determined a merge commit as the culprit; the863  latter happens if other changes depend on the culprit. But if the revert864  succeeds, it is worth building another kernel, as it validates the result of865  a bisection, which can easily deroute; it furthermore will let kernel866  developers know, if they can resolve the regression with a quick revert.867 868  Begin by checking out the latest codebase depending on the range you bisected:869 870  * Did you face a regression within a stable/longterm series (say between871    6.0.13 and 6.0.15) that does not happen in mainline? Then check out the872    latest codebase for the affected series like this::873 874      git fetch stable875      git switch --discard-changes --detach linux-6.0.y876 877  * In all other cases check out latest mainline::878 879      git fetch mainline880      git switch --discard-changes --detach mainline/master881 882    If you bisected a regression within a stable/longterm series that also883    happens in mainline, there is one more thing to do: look up the mainline884    commit-id. To do so, use a command like ``git show abcdcafecabcd`` to885    view the patch description of the culprit. There will be a line near886    the top which looks like 'commit cafec0cacaca0 upstream.' or887    'Upstream commit cafec0cacaca0'; use that commit-id in the next command888    and not the one the bisection blamed.889 890  Now try reverting the culprit by specifying its commit id::891 892    git revert --no-edit cafec0cacaca0893 894  If that fails, give up trying and move on to the next step; if it works,895  adjust the tag to facilitate the identification and prevent accidentally896  overwriting another kernel::897 898    cp ~/kernel-config-working .config899    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'900 901  Build a kernel using the familiar command sequence, just without copying the902  the base .config over::903 904    make olddefconfig &&905    make -j $(nproc --all)906    # * Check if the free space suffices holding another kernel:907    df -h /boot/ /lib/modules/908    sudo make modules_install909    command -v installkernel && sudo make install910    make -s kernelrelease | tee -a ~/kernels-built911    reboot912 913  Now check one last time if the feature that made you perform a bisection works914  with that kernel: if everything went well, it should not show the regression.915 916  [:ref:`details <revert_bisref>`]917 918.. _introclosure_bissbs:919 920Complementary tasks: cleanup during and after the bisection921-----------------------------------------------------------922 923During and after following this guide you might want or need to remove some of924the kernels you installed: the boot menu otherwise will become confusing or925space might run out.926 927.. _makeroom_bissbs:928 929* To remove one of the kernels you installed, look up its 'kernelrelease'930  identifier. This guide stores them in '~/kernels-built', but the following931  command will print them as well::932 933    ls -ltr /lib/modules/*-local*934 935  You in most situations want to remove the oldest kernels built during the936  actual bisection (e.g. segment 3 of this guide). The two ones you created937  beforehand (e.g. to test the latest codebase and the version considered938  'good') might become handy to verify something later -- thus better keep them939  around, unless you are really short on storage space.940 941  To remove the modules of a kernel with the kernelrelease identifier942  '*6.0-rc1-local-gcafec0cacaca0*', start by removing the directory holding its943  modules::944 945    sudo rm -rf /lib/modules/6.0-rc1-local-gcafec0cacaca0946 947  Afterwards try the following command::948 949    sudo kernel-install -v remove 6.0-rc1-local-gcafec0cacaca0950 951  On quite a few distributions this will delete all other kernel files installed952  while also removing the kernel's entry from the boot menu. But on some953  distributions kernel-install does not exist or leaves boot-loader entries or954  kernel image and related files behind; in that case remove them as described955  in the reference section.956 957  [:ref:`details <makeroom_bisref>`]958 959.. _finishingtouch_bissbs:960 961* Once you have finished the bisection, do not immediately remove anything you962  set up, as you might need a few things again. What is safe to remove depends963  on the outcome of the bisection:964 965  * Could you initially reproduce the regression with the latest codebase and966    after the bisection were able to fix the problem by reverting the culprit on967    top of the latest codebase? Then you want to keep those two kernels around968    for a while, but safely remove all others with a '-local' in the release969    identifier.970 971  * Did the bisection end on a merge-commit or seems questionable for other972    reasons? Then you want to keep as many kernels as possible around for a few973    days: it's pretty likely that you will be asked to recheck something.974 975  * In other cases it likely is a good idea to keep the following kernels around976    for some time: the one built from the latest codebase, the one created from977    the version considered 'good', and the last three or four you compiled978    during the actual bisection process.979 980  [:ref:`details <finishingtouch_bisref>`]981 982.. _introoptional_bissbs:983 984Optional: test reverts, patches, or later versions985--------------------------------------------------986 987While or after reporting a bug, you might want or potentially will be asked to988test reverts, debug patches, proposed fixes, or other versions. In that case989follow these instructions.990 991* Update your Git clone and check out the latest code.992 993  * In case you want to test mainline, fetch its latest changes before checking994    its code out::995 996      git fetch mainline997      git switch --discard-changes --detach mainline/master998 999  * In case you want to test a stable or longterm kernel, first add the branch1000    holding the series you are interested in (6.2 in the example), unless you1001    already did so earlier::1002 1003      git remote set-branches --add stable linux-6.2.y1004 1005    Then fetch the latest changes and check out the latest version from the1006    series::1007 1008      git fetch stable1009      git switch --discard-changes --detach stable/linux-6.2.y1010 1011* Copy your kernel build configuration over::1012 1013    cp ~/kernel-config-working .config1014 1015* Your next step depends on what you want to do:1016 1017  * In case you just want to test the latest codebase, head to the next step,1018    you are already all set.1019 1020  * In case you want to test if a revert fixes an issue, revert one or multiple1021    changes by specifying their commit ids::1022 1023      git revert --no-edit cafec0cacaca01024 1025    Now give that kernel a special tag to facilitates its identification and1026    prevent accidentally overwriting another kernel::1027 1028      ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted'1029 1030  * In case you want to test a patch, store the patch in a file like1031    '/tmp/foobars-proposed-fix-v1.patch' and apply it like this::1032 1033      git apply /tmp/foobars-proposed-fix-v1.patch1034 1035    In case of multiple patches, repeat this step with the others.1036 1037    Now give that kernel a special tag to facilitates its identification and1038    prevent accidentally overwriting another kernel::1039 1040    ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1'1041 1042* Build a kernel using the familiar commands, just without copying the kernel1043  build configuration over, as that has been taken care of already::1044 1045    make olddefconfig &&1046    make -j $(nproc --all)1047    # * Check if the free space suffices holding another kernel:1048    df -h /boot/ /lib/modules/1049    sudo make modules_install1050    command -v installkernel && sudo make install1051    make -s kernelrelease | tee -a ~/kernels-built1052    reboot1053 1054* Now verify you booted the newly built kernel and check it.1055 1056[:ref:`details <introoptional_bisref>`]1057 1058.. _submit_improvements:1059 1060Conclusion1061----------1062 1063You have reached the end of the step-by-step guide.1064 1065Did you run into trouble following any of the above steps not cleared up by the1066reference section below? Did you spot errors? Or do you have ideas how to1067improve the guide?1068 1069If any of that applies, please take a moment and let the maintainer of this1070document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while1071CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is1072vital to improve this text further, which is in everybody's interest, as it1073will enable more people to master the task described here -- and hopefully also1074improve similar guides inspired by this one.1075 1076 1077Reference section for the step-by-step guide1078============================================1079 1080This section holds additional information for almost all the items in the above1081step-by-step guide.1082 1083Preparations for building your own kernels1084------------------------------------------1085 1086  *The steps in this section lay the groundwork for all further tests.*1087  [:ref:`... <introprep_bissbs>`]1088 1089The steps in all later sections of this guide depend on those described here.1090 1091[:ref:`back to step-by-step guide <introprep_bissbs>`].1092 1093.. _backup_bisref:1094 1095Prepare for emergencies1096~~~~~~~~~~~~~~~~~~~~~~~1097 1098  *Create a fresh backup and put system repair and restore tools at hand.*1099  [:ref:`... <backup_bissbs>`]1100 1101Remember, you are dealing with computers, which sometimes do unexpected things1102-- especially if you fiddle with crucial parts like the kernel of an operating1103system. That's what you are about to do in this process. Hence, better prepare1104for something going sideways, even if that should not happen.1105 1106[:ref:`back to step-by-step guide <backup_bissbs>`]1107 1108.. _vanilla_bisref:1109 1110Remove anything related to externally maintained kernel modules1111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1112 1113  *Remove all software that depends on externally developed kernel drivers or1114  builds them automatically.* [:ref:`...<vanilla_bissbs>`]1115 1116Externally developed kernel modules can easily cause trouble during a bisection.1117 1118But there is a more important reason why this guide contains this step: most1119kernel developers will not care about reports about regressions occurring with1120kernels that utilize such modules. That's because such kernels are not1121considered 'vanilla' anymore, as Documentation/admin-guide/reporting-issues.rst1122explains in more detail.1123 1124[:ref:`back to step-by-step guide <vanilla_bissbs>`]1125 1126.. _secureboot_bisref:1127 1128Deal with techniques like Secure Boot1129~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1130 1131  *On platforms with 'Secure Boot' or similar techniques, prepare everything to1132  ensure the system will permit your self-compiled kernel to boot later.*1133  [:ref:`... <secureboot_bissbs>`]1134 1135Many modern systems allow only certain operating systems to start; that's why1136they reject booting self-compiled kernels by default.1137 1138You ideally deal with this by making your platform trust your self-built kernels1139with the help of a certificate. How to do that is not described1140here, as it requires various steps that would take the text too far away from1141its purpose; 'Documentation/admin-guide/module-signing.rst' and various web1142sides already explain everything needed in more detail.1143 1144Temporarily disabling solutions like Secure Boot is another way to make your own1145Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup1146utility; the required steps vary a lot between machines and therefore cannot be1147described here.1148 1149On mainstream x86 Linux distributions there is a third and universal option:1150disable all Secure Boot restrictions for your Linux environment. You can1151initiate this process by running ``mokutil --disable-validation``; this will1152tell you to create a one-time password, which is safe to write down. Now1153restart; right after your BIOS performed all self-tests the bootloader Shim will1154show a blue box with a message 'Press any key to perform MOK management'. Hit1155some key before the countdown exposes, which will open a menu. Choose 'Change1156Secure Boot state'. Shim's 'MokManager' will now ask you to enter three1157randomly chosen characters from the one-time password specified earlier. Once1158you provided them, confirm you really want to disable the validation.1159Afterwards, permit MokManager to reboot the machine.1160 1161[:ref:`back to step-by-step guide <secureboot_bissbs>`]1162 1163.. _bootworking_bisref:1164 1165Boot the last kernel that was working1166~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1167 1168  *Boot into the last working kernel and briefly recheck if the feature that1169  regressed really works.* [:ref:`...<bootworking_bissbs>`]1170 1171This will make later steps that cover creating and trimming the configuration do1172the right thing.1173 1174[:ref:`back to step-by-step guide <bootworking_bissbs>`]1175 1176.. _diskspace_bisref:1177 1178Space requirements1179~~~~~~~~~~~~~~~~~~1180 1181  *Ensure to have enough free space for building Linux.*1182  [:ref:`... <diskspace_bissbs>`]1183 1184The numbers mentioned are rough estimates with a big extra charge to be on the1185safe side, so often you will need less.1186 1187If you have space constraints, be sure to hay attention to the :ref:`step about1188debug symbols' <debugsymbols_bissbs>` and its :ref:`accompanying reference1189section' <debugsymbols_bisref>`, as disabling then will reduce the consumed disk1190space by quite a few gigabytes.1191 1192[:ref:`back to step-by-step guide <diskspace_bissbs>`]1193 1194.. _rangecheck_bisref:1195 1196Bisection range1197~~~~~~~~~~~~~~~1198 1199  *Determine the kernel versions considered 'good' and 'bad' throughout this1200  guide.* [:ref:`...<rangecheck_bissbs>`]1201 1202Establishing the range of commits to be checked is mostly straightforward,1203except when a regression occurred when switching from a release of one stable1204series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case1205Git will need some hand holding, as there is no straight line of descent.1206 1207That's because with the release of 6.0 mainline carried on to 6.1 while the1208stable series 6.0.y branched to the side. It's therefore theoretically possible1209that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a1210commit that went into one of the 6.0.y releases, but never hit mainline or the12116.1.y series. Thankfully that normally should not happen due to the way the1212stable/longterm maintainers maintain the code. It's thus pretty safe to assume12136.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel1214will be built and tested in the segment '2' of this guide; Git would force you1215to do this as well, if you tried bisecting between 6.0.13 and 6.1.15.1216 1217[:ref:`back to step-by-step guide <rangecheck_bissbs>`]1218 1219.. _buildrequires_bisref:1220 1221Install build requirements1222~~~~~~~~~~~~~~~~~~~~~~~~~~1223 1224  *Install all software required to build a Linux kernel.*1225  [:ref:`...<buildrequires_bissbs>`]1226 1227The kernel is pretty stand-alone, but besides tools like the compiler you will1228sometimes need a few libraries to build one. How to install everything needed1229depends on your Linux distribution and the configuration of the kernel you are1230about to build.1231 1232Here are a few examples what you typically need on some mainstream1233distributions:1234 1235* Arch Linux and derivatives::1236 1237    sudo pacman --needed -S bc binutils bison flex gcc git kmod libelf openssl \1238      pahole perl zlib ncurses qt6-base1239 1240* Debian, Ubuntu, and derivatives::1241 1242    sudo apt install bc binutils bison dwarves flex gcc git kmod libelf-dev \1243      libssl-dev make openssl pahole perl-base pkg-config zlib1g-dev \1244      libncurses-dev qt6-base-dev g++1245 1246* Fedora and derivatives::1247 1248    sudo dnf install binutils \1249      /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole,rpmbuild} \1250      /usr/include/{libelf.h,openssl/pkcs7.h,zlib.h,ncurses.h,qt6/QtGui/QAction}1251 1252* openSUSE and derivatives::1253 1254    sudo zypper install bc binutils bison dwarves flex gcc git \1255      kernel-install-tools libelf-devel make modutils openssl openssl-devel \1256      perl-base zlib-devel rpm-build ncurses-devel qt6-base-devel1257 1258These commands install a few packages that are often, but not always needed. You1259for example might want to skip installing the development headers for ncurses,1260which you will only need in case you later might want to adjust the kernel build1261configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit1262the headers of Qt6 if you do not plan to adjust the .config using 'xconfig'.1263 1264You furthermore might need additional libraries and their development headers1265for tasks not covered in this guide -- for example when building utilities from1266the kernel's tools/ directory.1267 1268[:ref:`back to step-by-step guide <buildrequires_bissbs>`]1269 1270.. _sources_bisref:1271 1272Download the sources using Git1273~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1274 1275  *Retrieve the Linux mainline sources.*1276  [:ref:`...<sources_bissbs>`]1277 1278The step-by-step guide outlines how to download the Linux sources using a full1279Git clone of Linus' mainline repository. There is nothing more to say about1280that -- but there are two alternatives ways to retrieve the sources that might1281work better for you:1282 1283* If you have an unreliable internet connection, consider1284  :ref:`using a 'Git bundle'<sources_bundle_bisref>`.1285 1286* If downloading the complete repository would take too long or requires too1287  much storage space, consider :ref:`using a 'shallow1288  clone'<sources_shallow_bisref>`.1289 1290.. _sources_bundle_bisref:1291 1292Downloading Linux mainline sources using a bundle1293"""""""""""""""""""""""""""""""""""""""""""""""""1294 1295Use the following commands to retrieve the Linux mainline sources using a1296bundle::1297 1298    wget -c \1299      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle1300    git clone --no-checkout clone.bundle ~/linux/1301    cd ~/linux/1302    git remote remove origin1303    git remote add mainline \1304      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git1305    git fetch mainline1306    git remote add -t master stable \1307      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git1308 1309In case the 'wget' command fails, just re-execute it, it will pick up where1310it left off.1311 1312[:ref:`back to step-by-step guide <sources_bissbs>`]1313[:ref:`back to section intro <sources_bisref>`]1314 1315.. _sources_shallow_bisref:1316 1317Downloading Linux mainline sources using a shallow clone1318~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1319 1320First, execute the following command to retrieve the latest mainline codebase::1321 1322    git clone -o mainline --no-checkout --depth 1 -b master \1323      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/1324    cd ~/linux/1325    git remote add -t master stable \1326      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git1327 1328Now deepen your clone's history to the second predecessor of the mainline1329release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would1330be the first predecessor and 5.18 the second -- hence deepen the history up to1331that version::1332 1333    git fetch --shallow-exclude=v5.18 mainline1334 1335Afterwards add the stable Git repository as remote and all required stable1336branches as explained in the step-by-step guide.1337 1338Note, shallow clones have a few peculiar characteristics:1339 1340* For bisections the history needs to be deepened a few mainline versions1341  farther than it seems necessary, as explained above already. That's because1342  Git otherwise will be unable to revert or describe most of the commits within1343  a range (say 6.1..6.2), as they are internally based on earlier kernels1344  releases (like 6.0-rc2 or 5.19-rc3).1345 1346* This document in most places uses ``git fetch`` with ``--shallow-exclude=``1347  to specify the earliest version you care about (or to be precise: its git1348  tag). You alternatively can use the parameter ``--shallow-since=`` to specify1349  an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to1350  define the depth of the history you want to download. When using them while1351  bisecting mainline, ensure to deepen the history to at least 7 months before1352  the release of the mainline release your 'good' kernel is based on.1353 1354* Be warned, when deepening your clone you might encounter an error like1355  'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'.1356  In that case run ``git repack -d`` and try again.1357 1358[:ref:`back to step-by-step guide <sources_bissbs>`]1359[:ref:`back to section intro <sources_bisref>`]1360 1361.. _oldconfig_bisref:1362 1363Start defining the build configuration for your kernel1364~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1365 1366  *Start preparing a kernel build configuration (the '.config' file).*1367  [:ref:`... <oldconfig_bissbs>`]1368 1369*Note, this is the first of multiple steps in this guide that create or modify1370build artifacts. The commands used in this guide store them right in the source1371tree to keep things simple. In case you prefer storing the build artifacts1372separately, create a directory like '~/linux-builddir/' and add the parameter1373``O=~/linux-builddir/`` to all make calls used throughout this guide. You will1374have to point other commands there as well -- among them the ``./scripts/config1375[...]`` commands, which will require ``--file ~/linux-builddir/.config`` to1376locate the right build configuration.*1377 1378Two things can easily go wrong when creating a .config file as advised:1379 1380* The oldconfig target will use a .config file from your build directory, if1381  one is already present there (e.g. '~/linux/.config'). That's totally fine if1382  that's what you intend (see next step), but in all other cases you want to1383  delete it. This for example is important in case you followed this guide1384  further, but due to problems come back here to redo the configuration from1385  scratch.1386 1387* Sometimes olddefconfig is unable to locate the .config file for your running1388  kernel and will use defaults, as briefly outlined in the guide. In that case1389  check if your distribution ships the configuration somewhere and manually put1390  it in the right place (e.g. '~/linux/.config') if it does. On distributions1391  where /proc/config.gz exists this can be achieved using this command::1392 1393    zcat /proc/config.gz > .config1394 1395  Once you put it there, run ``make olddefconfig`` again to adjust it to the1396  needs of the kernel about to be built.1397 1398Note, the olddefconfig target will set any undefined build options to their1399default value. If you prefer to set such configuration options manually, use1400``make oldconfig`` instead. Then for each undefined configuration option you1401will be asked how to proceed; in case you are unsure what to answer, simply hit1402'enter' to apply the default value. Note though that for bisections you normally1403want to go with the defaults, as you otherwise might enable a new feature that1404causes a problem looking like regressions (for example due to security1405restrictions).1406 1407Occasionally odd things happen when trying to use a config file prepared for one1408kernel (say 6.1) on an older mainline release -- especially if it is much older1409(say 5.15). That's one of the reasons why the previous step in the guide told1410you to boot the kernel where everything works. If you manually add a .config1411file you thus want to ensure it's from the working kernel and not from a one1412that shows the regression.1413 1414In case you want to build kernels for another machine, locate its kernel build1415configuration; usually ``ls /boot/config-$(uname -r)`` will print its name. Copy1416that file to the build machine and store it as ~/linux/.config; afterwards run1417``make olddefconfig`` to adjust it.1418 1419[:ref:`back to step-by-step guide <oldconfig_bissbs>`]1420 1421.. _localmodconfig_bisref:1422 1423Trim the build configuration for your kernel1424~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1425 1426  *Disable any kernel modules apparently superfluous for your setup.*1427  [:ref:`... <localmodconfig_bissbs>`]1428 1429As explained briefly in the step-by-step guide already: with localmodconfig it1430can easily happen that your self-built kernels will lack modules for tasks you1431did not perform at least once before utilizing this make target. That happens1432when a task requires kernel modules which are only autoloaded when you execute1433it for the first time. So when you never performed that task since starting your1434kernel the modules will not have been loaded -- and from localmodonfig's point1435of view look superfluous, which thus disables them to reduce the amount of code1436to be compiled.1437 1438You can try to avoid this by performing typical tasks that often will autoload1439additional kernel modules: start a VM, establish VPN connections, loop-mount a1440CD/DVD ISO, mount network shares (CIFS, NFS, ...), and connect all external1441devices (2FA keys, headsets, webcams, ...) as well as storage devices with file1442systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, ...). But it1443is hard to think of everything that might be needed -- even kernel developers1444often forget one thing or another at this point.1445 1446Do not let that risk bother you, especially when compiling a kernel only for1447testing purposes: everything typically crucial will be there. And if you forget1448something important you can turn on a missing feature manually later and quickly1449run the commands again to compile and install a kernel that has everything you1450need.1451 1452But if you plan to build and use self-built kernels regularly, you might want to1453reduce the risk by recording which modules your system loads over the course of1454a few weeks. You can automate this with `modprobed-db1455<https://github.com/graysky2/modprobed-db>`_. Afterwards use ``LSMOD=<path>`` to1456point localmodconfig to the list of modules modprobed-db noticed being used::1457 1458  yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig1459 1460That parameter also allows you to build trimmed kernels for another machine in1461case you copied a suitable .config over to use as base (see previous step). Just1462run ``lsmod > lsmod_foo-machine`` on that system and copy the generated file to1463your build's host home directory. Then run these commands instead of the one the1464step-by-step guide mentions::1465 1466  yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig1467 1468[:ref:`back to step-by-step guide <localmodconfig_bissbs>`]1469 1470.. _tagging_bisref:1471 1472Tag the kernels about to be build1473~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1474 1475  *Ensure all the kernels you will build are clearly identifiable using a1476  special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`]1477 1478This allows you to differentiate your distribution's kernels from those created1479during this process, as the file or directories for the latter will contain1480'-local' in the name; it also helps picking the right entry in the boot menu and1481not lose track of you kernels, as their version numbers will look slightly1482confusing during the bisection.1483 1484[:ref:`back to step-by-step guide <tagging_bissbs>`]1485 1486.. _debugsymbols_bisref:1487 1488Decide to enable or disable debug symbols1489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1490 1491  *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`]1492 1493Having debug symbols available can be important when your kernel throws a1494'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be1495able to find the exact place where the problem occurred in the code. But1496collecting and embedding the needed debug information takes time and consumes1497quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel1498trimmed with localmodconfig consumed around 5 Gigabyte of space with debug1499symbols, but less than 1 when they were disabled. The resulting kernel image and1500modules are bigger as well, which increases storage requirements for /boot/ and1501load times.1502 1503In case you want a small kernel and are unlikely to decode a stack trace later,1504you thus might want to disable debug symbols to avoid those downsides. If it1505later turns out that you need them, just enable them as shown and rebuild the1506kernel.1507 1508You on the other hand definitely want to enable them for this process, if there1509is a decent chance that you need to decode a stack trace later. The section1510'Decode failure messages' in Documentation/admin-guide/reporting-issues.rst1511explains this process in more detail.1512 1513[:ref:`back to step-by-step guide <debugsymbols_bissbs>`]1514 1515.. _configmods_bisref:1516 1517Adjust build configuration1518~~~~~~~~~~~~~~~~~~~~~~~~~~1519 1520  *Check if you may want or need to adjust some other kernel configuration1521  options:*1522 1523Depending on your needs you at this point might want or have to adjust some1524kernel configuration options.1525 1526.. _configmods_distros_bisref:1527 1528Distro specific adjustments1529"""""""""""""""""""""""""""1530 1531  *Are you running* [:ref:`... <configmods_bissbs>`]1532 1533The following sections help you to avoid build problems that are known to occur1534when following this guide on a few commodity distributions.1535 1536**Debian:**1537 1538* Remove a stale reference to a certificate file that would cause your build to1539  fail::1540 1541   ./scripts/config --set-str SYSTEM_TRUSTED_KEYS ''1542 1543  Alternatively, download the needed certificate and make that configuration1544  option point to it, as `the Debian handbook explains in more detail1545  <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_1546  -- or generate your own, as explained in1547  Documentation/admin-guide/module-signing.rst.1548 1549[:ref:`back to step-by-step guide <configmods_bissbs>`]1550 1551.. _configmods_individual_bisref:1552 1553Individual adjustments1554""""""""""""""""""""""1555 1556  *If you want to influence the other aspects of the configuration, do so1557  now.* [:ref:`... <configmods_bissbs>`]1558 1559At this point you can use a command like ``make menuconfig`` or ``make nconfig``1560to enable or disable certain features using a text-based user interface; to use1561a graphical configuration utility, run ``make xconfig`` instead. Both of them1562require development libraries from toolkits they are rely on (ncurses1563respectively Qt5 or Qt6); an error message will tell you if something required1564is missing.1565 1566[:ref:`back to step-by-step guide <configmods_bissbs>`]1567 1568.. _saveconfig_bisref:1569 1570Put the .config file aside1571~~~~~~~~~~~~~~~~~~~~~~~~~~1572 1573  *Reprocess the .config after the latest changes and store it in a safe place.*1574  [:ref:`... <saveconfig_bissbs>`]1575 1576Put the .config you prepared aside, as you want to copy it back to the build1577directory every time during this guide before you start building another1578kernel. That's because going back and forth between different versions can alter1579.config files in odd ways; those occasionally cause side effects that could1580confuse testing or in some cases render the result of your bisection1581meaningless.1582 1583[:ref:`back to step-by-step guide <saveconfig_bissbs>`]1584 1585.. _introlatestcheck_bisref:1586 1587Try to reproduce the problem with the latest codebase1588-----------------------------------------------------1589 1590  *Verify the regression is not caused by some .config change and check if it1591  still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`]1592 1593For some readers it might seem unnecessary to check the latest codebase at this1594point, especially if you did that already with a kernel prepared by your1595distributor or face a regression within a stable/longterm series. But it's1596highly recommended for these reasons:1597 1598* You will run into any problems caused by your setup before you actually begin1599  a bisection. That will make it a lot easier to differentiate between 'this1600  most likely is some problem in my setup' and 'this change needs to be skipped1601  during the bisection, as the kernel sources at that stage contain an unrelated1602  problem that causes building or booting to fail'.1603 1604* These steps will rule out if your problem is caused by some change in the1605  build configuration between the 'working' and the 'broken' kernel. This for1606  example can happen when your distributor enabled an additional security1607  feature in the newer kernel which was disabled or not yet supported by the1608  older kernel. That security feature might get into the way of something you1609  do -- in which case your problem from the perspective of the Linux kernel1610  upstream developers is not a regression, as1611  Documentation/admin-guide/reporting-regressions.rst explains in more detail.1612  You thus would waste your time if you'd try to bisect this.1613 1614* If the cause for your regression was already fixed in the latest mainline1615  codebase, you'd perform the bisection for nothing. This holds true for a1616  regression you encountered with a stable/longterm release as well, as they are1617  often caused by problems in mainline changes that were backported -- in which1618  case the problem will have to be fixed in mainline first. Maybe it already was1619  fixed there and the fix is already in the process of being backported.1620 1621* For regressions within a stable/longterm series it's furthermore crucial to1622  know if the issue is specific to that series or also happens in the mainline1623  kernel, as the report needs to be sent to different people:1624 1625  * Regressions specific to a stable/longterm series are the stable team's1626    responsibility; mainline Linux developers might or might not care.1627 1628  * Regressions also happening in mainline are something the regular Linux1629    developers and maintainers have to handle; the stable team does not care1630    and does not need to be involved in the report, they just should be told1631    to backport the fix once it's ready.1632 1633  Your report might be ignored if you send it to the wrong party -- and even1634  when you get a reply there is a decent chance that developers tell you to1635  evaluate which of the two cases it is before they take a closer look.1636 1637[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`]1638 1639.. _checkoutmaster_bisref:1640 1641Check out the latest Linux codebase1642~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1643 1644  *Check out the latest Linux codebase.*1645  [:ref:`... <checkoutmaster_bissbs>`]1646 1647In case you later want to recheck if an ever newer codebase might fix the1648problem, remember to run that ``git fetch --shallow-exclude [...]`` command1649again mentioned earlier to update your local Git repository.1650 1651[:ref:`back to step-by-step guide <checkoutmaster_bissbs>`]1652 1653.. _build_bisref:1654 1655Build your kernel1656~~~~~~~~~~~~~~~~~1657 1658  *Build the image and the modules of your first kernel using the config file1659  you prepared.* [:ref:`... <build_bissbs>`]1660 1661A lot can go wrong at this stage, but the instructions below will help you help1662yourself. Another subsection explains how to directly package your kernel up as1663deb, rpm or tar file.1664 1665Dealing with build errors1666"""""""""""""""""""""""""1667 1668When a build error occurs, it might be caused by some aspect of your machine's1669setup that often can be fixed quickly; other times though the problem lies in1670the code and can only be fixed by a developer. A close examination of the1671failure messages coupled with some research on the internet will often tell you1672which of the two it is. To perform such investigation, restart the build1673process like this::1674 1675  make V=11676 1677The ``V=1`` activates verbose output, which might be needed to see the actual1678error. To make it easier to spot, this command also omits the ``-j $(nproc1679--all)`` used earlier to utilize every CPU core in the system for the job -- but1680this parallelism also results in some clutter when failures occur.1681 1682After a few seconds the build process should run into the error again. Now try1683to find the most crucial line describing the problem. Then search the internet1684for the most important and non-generic section of that line (say 4 to 8 words);1685avoid or remove anything that looks remotely system-specific, like your username1686or local path names like ``/home/username/linux/``. First try your regular1687internet search engine with that string, afterwards search Linux kernel mailing1688lists via `lore.kernel.org/all/ <https://lore.kernel.org/all/>`_.1689 1690This most of the time will find something that will explain what is wrong; quite1691often one of the hits will provide a solution for your problem, too. If you1692do not find anything that matches your problem, try again from a different angle1693by modifying your search terms or using another line from the error messages.1694 1695In the end, most issues you run into have likely been encountered and1696reported by others already. That includes issues where the cause is not your1697system, but lies in the code. If you run into one of those, you might thus find1698a solution (e.g. a patch) or workaround for your issue, too.1699 1700Package your kernel up1701""""""""""""""""""""""1702 1703The step-by-step guide uses the default make targets (e.g. 'bzImage' and1704'modules' on x86) to build the image and the modules of your kernel, which later1705steps of the guide then install. You instead can also directly build everything1706and directly package it up by using one of the following targets:1707 1708* ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package1709 1710* ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package1711 1712* ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball1713 1714This is just a selection of available make targets for this purpose, see1715``make help`` for others. You can also use these targets after running1716``make -j $(nproc --all)``, as they will pick up everything already built.1717 1718If you employ the targets to generate deb or rpm packages, ignore the1719step-by-step guide's instructions on installing and removing your kernel;1720instead install and remove the packages using the package utility for the format1721(e.g. dpkg and rpm) or a package management utility build on top of them (apt,1722aptitude, dnf/yum, zypper, ...). Be aware that the packages generated using1723these two make targets are designed to work on various distributions utilizing1724those formats, they thus will sometimes behave differently than your1725distribution's kernel packages.1726 1727[:ref:`back to step-by-step guide <build_bissbs>`]1728 1729.. _install_bisref:1730 1731Put the kernel in place1732~~~~~~~~~~~~~~~~~~~~~~~1733 1734  *Install the kernel you just built.* [:ref:`... <install_bissbs>`]1735 1736What you need to do after executing the command in the step-by-step guide1737depends on the existence and the implementation of ``/sbin/installkernel``1738executable on your distribution.1739 1740If installkernel is found, the kernel's build system will delegate the actual1741installation of your kernel image to this executable, which then performs some1742or all of these tasks:1743 1744* On almost all Linux distributions installkernel will store your kernel's1745  image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will1746  put a 'System.map-<kernelrelease_id>' alongside it.1747 1748* On most distributions installkernel will then generate an 'initramfs'1749  (sometimes also called 'initrd'), which usually are stored as1750  '/boot/initramfs-<kernelrelease_id>.img' or1751  '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file1752  for booting, hence ensure to execute the make target 'modules_install' first,1753  as your distribution's initramfs generator otherwise will be unable to find1754  the modules that go into the image.1755 1756* On some distributions installkernel will then add an entry for your kernel1757  to your bootloader's configuration.1758 1759You have to take care of some or all of the tasks yourself, if your1760distribution lacks a installkernel script or does only handle part of them.1761Consult the distribution's documentation for details. If in doubt, install the1762kernel manually::1763 1764   sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)1765   sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)1766 1767Now generate your initramfs using the tools your distribution provides for this1768process. Afterwards add your kernel to your bootloader configuration and reboot.1769 1770[:ref:`back to step-by-step guide <install_bissbs>`]1771 1772.. _storagespace_bisref:1773 1774Storage requirements per kernel1775~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1776 1777  *Check how much storage space the kernel, its modules, and other related files1778  like the initramfs consume.* [:ref:`... <storagespace_bissbs>`]1779 1780The kernels built during a bisection consume quite a bit of space in /boot/ and1781/lib/modules/, especially if you enabled debug symbols. That makes it easy to1782fill up volumes during a bisection -- and due to that even kernels which used to1783work earlier might fail to boot. To prevent that you will need to know how much1784space each installed kernel typically requires.1785 1786Note, most of the time the pattern '/boot/*$(make -s kernelrelease)*' used in1787the guide will match all files needed to boot your kernel -- but neither the1788path nor the naming scheme are mandatory. On some distributions you thus will1789need to look in different places.1790 1791[:ref:`back to step-by-step guide <storagespace_bissbs>`]1792 1793.. _tainted_bisref:1794 1795Check if your newly built kernel considers itself 'tainted'1796~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1797 1798  *Check if the kernel marked itself as 'tainted'.*1799  [:ref:`... <tainted_bissbs>`]1800 1801Linux marks itself as tainted when something happens that potentially leads to1802follow-up errors that look totally unrelated. That is why developers might1803ignore or react scantly to reports from tainted kernels -- unless of course the1804kernel set the flag right when the reported bug occurred.1805 1806That's why you want check why a kernel is tainted as explained in1807Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own1808interest, as your testing might be flawed otherwise.1809 1810[:ref:`back to step-by-step guide <tainted_bissbs>`]1811 1812.. _recheckbroken_bisref:1813 1814Check the kernel built from a recent mainline codebase1815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1816 1817  *Verify if your bug occurs with the newly built kernel.*1818  [:ref:`... <recheckbroken_bissbs>`]1819 1820There are a couple of reasons why your bug or regression might not show up with1821the kernel you built from the latest codebase. These are the most frequent:1822 1823* The bug was fixed meanwhile.1824 1825* What you suspected to be a regression was caused by a change in the build1826  configuration the provider of your kernel carried out.1827 1828* Your problem might be a race condition that does not show up with your kernel;1829  the trimmed build configuration, a different setting for debug symbols, the1830  compiler used, and various other things can cause this.1831 1832* In case you encountered the regression with a stable/longterm kernel it might1833  be a problem that is specific to that series; the next step in this guide will1834  check this.1835 1836[:ref:`back to step-by-step guide <recheckbroken_bissbs>`]1837 1838.. _recheckstablebroken_bisref:1839 1840Check the kernel built from the latest stable/longterm codebase1841~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1842 1843  *Are you facing a regression within a stable/longterm release, but failed to1844  reproduce it with the kernel you just built using the latest mainline sources?1845  Then check if the latest codebase for the particular series might already fix1846  the problem.* [:ref:`... <recheckstablebroken_bissbs>`]1847 1848If this kernel does not show the regression either, there most likely is no need1849for a bisection.1850 1851[:ref:`back to step-by-step guide <recheckstablebroken_bissbs>`]1852 1853.. _introworkingcheck_bisref:1854 1855Ensure the 'good' version is really working well1856------------------------------------------------1857 1858  *Check if the kernels you build work fine.*1859  [:ref:`... <introworkingcheck_bissbs>`]1860 1861This section will reestablish a known working base. Skipping it might be1862appealing, but is usually a bad idea, as it does something important:1863 1864It will ensure the .config file you prepared earlier actually works as expected.1865That is in your own interest, as trimming the configuration is not foolproof --1866and you might be building and testing ten or more kernels for nothing before1867starting to suspect something might be wrong with the build configuration.1868 1869That alone is reason enough to spend the time on this, but not the only reason.1870 1871Many readers of this guide normally run kernels that are patched, use add-on1872modules, or both. Those kernels thus are not considered 'vanilla' -- therefore1873it's possible that the thing that regressed might never have worked in vanilla1874builds of the 'good' version in the first place.1875 1876There is a third reason for those that noticed a regression between1877stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will1878ensure the kernel version you assumed to be 'good' earlier in the process (e.g.18796.0) actually is working.1880 1881[:ref:`back to step-by-step guide <introworkingcheck_bissbs>`]1882 1883.. _recheckworking_bisref:1884 1885Build your own version of the 'good' kernel1886~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1887 1888  *Build your own variant of the working kernel and check if the feature that1889  regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`]1890 1891In case the feature that broke with newer kernels does not work with your first1892self-built kernel, find and resolve the cause before moving on. There are a1893multitude of reasons why this might happen. Some ideas where to look:1894 1895* Check the taint status and the output of ``dmesg``, maybe something unrelated1896  went wrong.1897 1898* Maybe localmodconfig did something odd and disabled the module required to1899  test the feature? Then you might want to recreate a .config file based on the1900  one from the last working kernel and skip trimming it down; manually disabling1901  some features in the .config might work as well to reduce the build time.1902 1903* Maybe it's not a kernel regression and something that is caused by some fluke,1904  a broken initramfs (also known as initrd), new firmware files, or an updated1905  userland software?1906 1907* Maybe it was a feature added to your distributor's kernel which vanilla Linux1908  at that point never supported?1909 1910Note, if you found and fixed problems with the .config file, you want to use it1911to build another kernel from the latest codebase, as your earlier tests with1912mainline and the latest version from an affected stable/longterm series were1913most likely flawed.1914 1915[:ref:`back to step-by-step guide <recheckworking_bissbs>`]1916 1917Perform a bisection and validate the result1918-------------------------------------------1919 1920  *With all the preparations and precaution builds taken care of, you are now1921  ready to begin the bisection.* [:ref:`... <introbisect_bissbs>`]1922 1923The steps in this segment perform and validate the bisection.1924 1925[:ref:`back to step-by-step guide <introbisect_bissbs>`].1926 1927.. _bisectstart_bisref:1928 1929Start the bisection1930~~~~~~~~~~~~~~~~~~~1931 1932  *Start the bisection and tell Git about the versions earlier established as1933  'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`]1934 1935This will start the bisection process; the last of the commands will make Git1936check out a commit round about half-way between the 'good' and the 'bad' changes1937for you to test.1938 1939[:ref:`back to step-by-step guide <bisectstart_bissbs>`]1940 1941.. _bisectbuild_bisref:1942 1943Build a kernel from the bisection point1944~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1945 1946  *Build, install, and boot a kernel from the code Git checked out using the1947  same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`]1948 1949There are two things worth of note here:1950 1951* Occasionally building the kernel will fail or it might not boot due some1952  problem in the code at the bisection point. In that case run this command::1953 1954    git bisect skip1955 1956  Git will then check out another commit nearby which with a bit of luck should1957  work better. Afterwards restart executing this step.1958 1959* Those slightly odd looking version identifiers can happen during bisections,1960  because the Linux kernel subsystems prepare their changes for a new mainline1961  release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus1962  base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then1963  get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This1964  leads to those slightly odd looking version identifiers coming up during1965  bisections.1966 1967[:ref:`back to step-by-step guide <bisectbuild_bissbs>`]1968 1969.. _bisecttest_bisref:1970 1971Bisection checkpoint1972~~~~~~~~~~~~~~~~~~~~1973 1974  *Check if the feature that regressed works in the kernel you just built.*1975  [:ref:`... <bisecttest_bissbs>`]1976 1977Ensure what you tell Git is accurate: getting it wrong just one time will bring1978the rest of the bisection totally off course, hence all testing after that point1979will be for nothing.1980 1981[:ref:`back to step-by-step guide <bisecttest_bissbs>`]1982 1983.. _bisectlog_bisref:1984 1985Put the bisection log away1986~~~~~~~~~~~~~~~~~~~~~~~~~~1987 1988  *Store Git's bisection log and the current .config file in a safe place.*1989  [:ref:`... <bisectlog_bissbs>`]1990 1991As indicated above: declaring just one kernel wrongly as 'good' or 'bad' will1992render the end result of a bisection useless. In that case you'd normally have1993to restart the bisection from scratch. The log can prevent that, as it might1994allow someone to point out where a bisection likely went sideways -- and then1995instead of testing ten or more kernels you might only have to build a few to1996resolve things.1997 1998The .config file is put aside, as there is a decent chance that developers might1999ask for it after you report the regression.2000 2001[:ref:`back to step-by-step guide <bisectlog_bissbs>`]2002 2003.. _revert_bisref:2004 2005Try reverting the culprit2006~~~~~~~~~~~~~~~~~~~~~~~~~2007 2008  *Try reverting the culprit on top of the latest codebase to see if this fixes2009  your regression.* [:ref:`... <revert_bissbs>`]2010 2011This is an optional step, but whenever possible one you should try: there is a2012decent chance that developers will ask you to perform this step when you bring2013the bisection result up. So give it a try, you are in the flow already, building2014one more kernel shouldn't be a big deal at this point.2015 2016The step-by-step guide covers everything relevant already except one slightly2017rare thing: did you bisected a regression that also happened with mainline using2018a stable/longterm series, but Git failed to revert the commit in mainline? Then2019try to revert the culprit in the affected stable/longterm series -- and if that2020succeeds, test that kernel version instead.2021 2022[:ref:`back to step-by-step guide <revert_bissbs>`]2023 2024Cleanup steps during and after following this guide2025---------------------------------------------------2026 2027  *During and after following this guide you might want or need to remove some2028  of the kernels you installed.* [:ref:`... <introclosure_bissbs>`]2029 2030The steps in this section describe clean-up procedures.2031 2032[:ref:`back to step-by-step guide <introclosure_bissbs>`].2033 2034.. _makeroom_bisref:2035 2036Cleaning up during the bisection2037~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~2038 2039  *To remove one of the kernels you installed, look up its 'kernelrelease'2040  identifier.* [:ref:`... <makeroom_bissbs>`]2041 2042The kernels you install during this process are easy to remove later, as its2043parts are only stored in two places and clearly identifiable. You thus do not2044need to worry to mess up your machine when you install a kernel manually (and2045thus bypass your distribution's packaging system): all parts of your kernels are2046relatively easy to remove later.2047 2048One of the two places is a directory in /lib/modules/, which holds the modules2049for each installed kernel. This directory is named after the kernel's release2050identifier; hence, to remove all modules for one of the kernels you built,2051simply remove its modules directory in /lib/modules/.2052 2053The other place is /boot/, where typically two up to five files will be placed2054during installation of a kernel. All of them usually contain the release name in2055their file name, but how many files and their exact names depend somewhat on2056your distribution's installkernel executable and its initramfs generator. On2057some distributions the ``kernel-install remove...`` command mentioned in the2058step-by-step guide will delete all of these files for you while also removing2059the menu entry for the kernel from your bootloader configuration. On others you2060have to take care of these two tasks yourself. The following command should2061interactively remove the three main files of a kernel with the release name2062'6.0-rc1-local-gcafec0cacaca0'::2063 2064  rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca02065 2066Afterwards check for other files in /boot/ that have2067'6.0-rc1-local-gcafec0cacaca0' in their name and consider deleting them as well.2068Now remove the boot entry for the kernel from your bootloader's configuration;2069the steps to do that vary quite a bit between Linux distributions.2070 2071Note, be careful with wildcards like '*' when deleting files or directories2072for kernels manually: you might accidentally remove files of a 6.0.13 kernel2073when all you want is to remove 6.0 or 6.0.1.2074 2075[:ref:`back to step-by-step guide <makeroom_bissbs>`]2076 2077Cleaning up after the bisection2078~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~2079 2080.. _finishingtouch_bisref:2081 2082  *Once you have finished the bisection, do not immediately remove anything2083  you set up, as you might need a few things again.*2084  [:ref:`... <finishingtouch_bissbs>`]2085 2086When you are really short of storage space removing the kernels as described in2087the step-by-step guide might not free as much space as you would like. In that2088case consider running ``rm -rf ~/linux/*`` as well now. This will remove the2089build artifacts and the Linux sources, but will leave the Git repository2090(~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the2091sources back.2092 2093Removing the repository as well would likely be unwise at this point: there2094is a decent chance developers will ask you to build another kernel to2095perform additional tests -- like testing a debug patch or a proposed fix.2096Details on how to perform those can be found in the section :ref:`Optional2097tasks: test reverts, patches, or later versions <introoptional_bissbs>`.2098 2099Additional tests are also the reason why you want to keep the2100~/kernel-config-working file around for a few weeks.2101 2102[:ref:`back to step-by-step guide <finishingtouch_bissbs>`]2103 2104.. _introoptional_bisref:2105 2106Test reverts, patches, or later versions2107----------------------------------------2108 2109  *While or after reporting a bug, you might want or potentially will be asked2110  to test reverts, patches, proposed fixes, or other versions.*2111  [:ref:`... <introoptional_bissbs>`]2112 2113All the commands used in this section should be pretty straight forward, so2114there is not much to add except one thing: when setting a kernel tag as2115instructed, ensure it is not much longer than the one used in the example, as2116problems will arise if the kernelrelease identifier exceeds 63 characters.2117 2118[:ref:`back to step-by-step guide <introoptional_bissbs>`].2119 2120 2121Additional information2122======================2123 2124.. _buildhost_bis:2125 2126Build kernels on a different machine2127------------------------------------2128 2129To compile kernels on another system, slightly alter the step-by-step guide's2130instructions:2131 2132* Start following the guide on the machine where you want to install and test2133  the kernels later.2134 2135* After executing ':ref:`Boot into the working kernel and briefly use the2136  apparently broken feature <bootworking_bissbs>`', save the list of loaded2137  modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the2138  build configuration for the running kernel (see ':ref:`Start defining the2139  build configuration for your kernel <oldconfig_bisref>`' for hints on where2140  to find it) and store it as '~/test-machine-config-working'. Transfer both2141  files to the home directory of your build host.2142 2143* Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough2144  free space for building [...] <diskspace_bissbs>`').2145 2146* When you reach ':ref:`Start preparing a kernel build configuration[...]2147  <oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time,2148  execute the following command to base your configuration on the one from the2149  test machine's 'working' kernel::2150 2151    cp ~/test-machine-config-working ~/linux/.config2152 2153* During the next step to ':ref:`disable any apparently superfluous kernel2154  modules <localmodconfig_bissbs>`' use the following command instead::2155 2156    yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig2157 2158* Continue the guide, but ignore the instructions outlining how to compile,2159  install, and reboot into a kernel every time they come up. Instead build2160  like this::2161 2162    cp ~/kernel-config-working .config2163    make olddefconfig &&2164    make -j $(nproc --all) targz-pkg2165 2166  This will generate a gzipped tar file whose name is printed in the last2167  line shown; for example, a kernel with the kernelrelease identifier2168  '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will2169  be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'.2170 2171  Copy that file to your test machine's home directory.2172 2173* Switch to the test machine to check if you have enough space to hold another2174  kernel. Then extract the file you transferred::2175 2176    sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C /2177 2178  Afterwards :ref:`generate the initramfs and add the kernel to your boot2179  loader's configuration <install_bisref>`; on some distributions the following2180  command will take care of both these tasks::2181 2182    sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa4232183 2184  Now reboot and ensure you started the intended kernel.2185 2186This approach even works when building for another architecture: just install2187cross-compilers and add the appropriate parameters to every invocation of make2188(e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``).2189 2190Additional reading material2191---------------------------2192 2193* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and2194  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_2195  in the Git documentation.2196* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_2197  from kernel developer Nathan Chancellor.2198* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.2199* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.2200 2201..2202   end-of-content2203..2204   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If2205   you spot a typo or small mistake, feel free to let him know directly and2206   he'll fix it. You are free to do the same in a mostly informal way if you2207   want to contribute changes to the text -- but for copyright reasons please CC2208   linux-doc@vger.kernel.org and 'sign-off' your contribution as2209   Documentation/process/submitting-patches.rst explains in the section 'Sign2210   your work - the Developer's Certificate of Origin'.2211..2212   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top2213   of the file. If you want to distribute this text under CC-BY-4.0 only,2214   please use 'The Linux kernel development community' for author attribution2215   and link this as source:2216   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst2217 2218..2219   Note: Only the content of this RST file as found in the Linux kernel sources2220   is available under CC-BY-4.0, as versions of this text that were processed2221   (for example by the kernel's build system) might contain content taken from2222   files which use a more restrictive license.2223