498 lines · plain
1.. _development_process:2 3How the development process works4=================================5 6Linux kernel development in the early 1990's was a pretty loose affair,7with relatively small numbers of users and developers involved. With a8user base in the millions and with some 2,000 developers involved over the9course of one year, the kernel has since had to evolve a number of10processes to keep development happening smoothly. A solid understanding of11how the process works is required in order to be an effective part of it.12 13The big picture14---------------15 16The kernel developers use a loosely time-based release process, with a new17major kernel release happening every two or three months. The recent18release history looks like this:19 20 ====== =================21 5.0 March 3, 201922 5.1 May 5, 201923 5.2 July 7, 201924 5.3 September 15, 201925 5.4 November 24, 201926 5.5 January 6, 202027 ====== =================28 29Every 5.x release is a major kernel release with new features, internal30API changes, and more. A typical release can contain about 13,00031changesets with changes to several hundred thousand lines of code. 5.x is32the leading edge of Linux kernel development; the kernel uses a33rolling development model which is continually integrating major changes.34 35A relatively straightforward discipline is followed with regard to the36merging of patches for each release. At the beginning of each development37cycle, the "merge window" is said to be open. At that time, code which is38deemed to be sufficiently stable (and which is accepted by the development39community) is merged into the mainline kernel. The bulk of changes for a40new development cycle (and all of the major changes) will be merged during41this time, at a rate approaching 1,000 changes ("patches," or "changesets")42per day.43 44(As an aside, it is worth noting that the changes integrated during the45merge window do not come out of thin air; they have been collected, tested,46and staged ahead of time. How that process works will be described in47detail later on).48 49The merge window lasts for approximately two weeks. At the end of this50time, Linus Torvalds will declare that the window is closed and release the51first of the "rc" kernels. For the kernel which is destined to be 5.6,52for example, the release which happens at the end of the merge window will53be called 5.6-rc1. The -rc1 release is the signal that the time to54merge new features has passed, and that the time to stabilize the next55kernel has begun.56 57Over the next six to ten weeks, only patches which fix problems should be58submitted to the mainline. On occasion a more significant change will be59allowed, but such occasions are rare; developers who try to merge new60features outside of the merge window tend to get an unfriendly reception.61As a general rule, if you miss the merge window for a given feature, the62best thing to do is to wait for the next development cycle. (An occasional63exception is made for drivers for previously-unsupported hardware; if they64touch no in-tree code, they cannot cause regressions and should be safe to65add at any time).66 67As fixes make their way into the mainline, the patch rate will slow over68time. Linus releases new -rc kernels about once a week; a normal series69will get up to somewhere between -rc6 and -rc9 before the kernel is70considered to be sufficiently stable and the final release is made.71At that point the whole process starts over again.72 73As an example, here is how the 5.4 development cycle went (all dates in742019):75 76 ============== ===============================77 September 15 5.3 stable release78 September 30 5.4-rc1, merge window closes79 October 6 5.4-rc280 October 13 5.4-rc381 October 20 5.4-rc482 October 27 5.4-rc583 November 3 5.4-rc684 November 10 5.4-rc785 November 17 5.4-rc886 November 24 5.4 stable release87 ============== ===============================88 89How do the developers decide when to close the development cycle and create90the stable release? The most significant metric used is the list of91regressions from previous releases. No bugs are welcome, but those which92break systems which worked in the past are considered to be especially93serious. For this reason, patches which cause regressions are looked upon94unfavorably and are quite likely to be reverted during the stabilization95period.96 97The developers' goal is to fix all known regressions before the stable98release is made. In the real world, this kind of perfection is hard to99achieve; there are just too many variables in a project of this size.100There comes a point where delaying the final release just makes the problem101worse; the pile of changes waiting for the next merge window will grow102larger, creating even more regressions the next time around. So most 5.x103kernels go out with a handful of known regressions though, hopefully, none104of them are serious.105 106Once a stable release is made, its ongoing maintenance is passed off to the107"stable team," currently Greg Kroah-Hartman. The stable team will release108occasional updates to the stable release using the 5.x.y numbering scheme.109To be considered for an update release, a patch must (1) fix a significant110bug, and (2) already be merged into the mainline for the next development111kernel. Kernels will typically receive stable updates for a little more112than one development cycle past their initial release. So, for example, the1135.2 kernel's history looked like this (all dates in 2019):114 115 ============== ===============================116 July 7 5.2 stable release117 July 14 5.2.1118 July 21 5.2.2119 July 26 5.2.3120 July 28 5.2.4121 July 31 5.2.5122 ... ...123 October 11 5.2.21124 ============== ===============================125 1265.2.21 was the final stable update of the 5.2 release.127 128Some kernels are designated "long term" kernels; they will receive support129for a longer period. Please refer to the following link for the list of active130long term kernel versions and their maintainers:131 132 https://www.kernel.org/category/releases.html133 134The selection of a kernel for long-term support is purely a matter of a135maintainer having the need and the time to maintain that release. There136are no known plans for long-term support for any specific upcoming137release.138 139 140The lifecycle of a patch141------------------------142 143Patches do not go directly from the developer's keyboard into the mainline144kernel. There is, instead, a somewhat involved (if somewhat informal)145process designed to ensure that each patch is reviewed for quality and that146each patch implements a change which is desirable to have in the mainline.147This process can happen quickly for minor fixes, or, in the case of large148and controversial changes, go on for years. Much developer frustration149comes from a lack of understanding of this process or from attempts to150circumvent it.151 152In the hopes of reducing that frustration, this document will describe how153a patch gets into the kernel. What follows below is an introduction which154describes the process in a somewhat idealized way. A much more detailed155treatment will come in later sections.156 157The stages that a patch goes through are, generally:158 159 - Design. This is where the real requirements for the patch - and the way160 those requirements will be met - are laid out. Design work is often161 done without involving the community, but it is better to do this work162 in the open if at all possible; it can save a lot of time redesigning163 things later.164 165 - Early review. Patches are posted to the relevant mailing list, and166 developers on that list reply with any comments they may have. This167 process should turn up any major problems with a patch if all goes168 well.169 170 - Wider review. When the patch is getting close to ready for mainline171 inclusion, it should be accepted by a relevant subsystem maintainer -172 though this acceptance is not a guarantee that the patch will make it173 all the way to the mainline. The patch will show up in the maintainer's174 subsystem tree and into the -next trees (described below). When the175 process works, this step leads to more extensive review of the patch and176 the discovery of any problems resulting from the integration of this177 patch with work being done by others.178 179- Please note that most maintainers also have day jobs, so merging180 your patch may not be their highest priority. If your patch is181 getting feedback about changes that are needed, you should either182 make those changes or justify why they should not be made. If your183 patch has no review complaints but is not being merged by its184 appropriate subsystem or driver maintainer, you should be persistent185 in updating the patch to the current kernel so that it applies cleanly186 and keep sending it for review and merging.187 188 - Merging into the mainline. Eventually, a successful patch will be189 merged into the mainline repository managed by Linus Torvalds. More190 comments and/or problems may surface at this time; it is important that191 the developer be responsive to these and fix any issues which arise.192 193 - Stable release. The number of users potentially affected by the patch194 is now large, so, once again, new problems may arise.195 196 - Long-term maintenance. While it is certainly possible for a developer197 to forget about code after merging it, that sort of behavior tends to198 leave a poor impression in the development community. Merging code199 eliminates some of the maintenance burden, in that others will fix200 problems caused by API changes. But the original developer should201 continue to take responsibility for the code if it is to remain useful202 in the longer term.203 204One of the largest mistakes made by kernel developers (or their employers)205is to try to cut the process down to a single "merging into the mainline"206step. This approach invariably leads to frustration for everybody207involved.208 209How patches get into the Kernel210-------------------------------211 212There is exactly one person who can merge patches into the mainline kernel213repository: Linus Torvalds. But, for example, of the over 9,500 patches214which went into the 2.6.38 kernel, only 112 (around 1.3%) were directly215chosen by Linus himself. The kernel project has long since grown to a size216where no single developer could possibly inspect and select every patch217unassisted. The way the kernel developers have addressed this growth is218through the use of a lieutenant system built around a chain of trust.219 220The kernel code base is logically broken down into a set of subsystems:221networking, specific architecture support, memory management, video222devices, etc. Most subsystems have a designated maintainer, a developer223who has overall responsibility for the code within that subsystem. These224subsystem maintainers are the gatekeepers (in a loose way) for the portion225of the kernel they manage; they are the ones who will (usually) accept a226patch for inclusion into the mainline kernel.227 228Subsystem maintainers each manage their own version of the kernel source229tree, usually (but certainly not always) using the git source management230tool. Tools like git (and related tools like quilt or mercurial) allow231maintainers to track a list of patches, including authorship information232and other metadata. At any given time, the maintainer can identify which233patches in his or her repository are not found in the mainline.234 235When the merge window opens, top-level maintainers will ask Linus to "pull"236the patches they have selected for merging from their repositories. If237Linus agrees, the stream of patches will flow up into his repository,238becoming part of the mainline kernel. The amount of attention that Linus239pays to specific patches received in a pull operation varies. It is clear240that, sometimes, he looks quite closely. But, as a general rule, Linus241trusts the subsystem maintainers to not send bad patches upstream.242 243Subsystem maintainers, in turn, can pull patches from other maintainers.244For example, the networking tree is built from patches which accumulated245first in trees dedicated to network device drivers, wireless networking,246etc. This chain of repositories can be arbitrarily long, though it rarely247exceeds two or three links. Since each maintainer in the chain trusts248those managing lower-level trees, this process is known as the "chain of249trust."250 251Clearly, in a system like this, getting patches into the kernel depends on252finding the right maintainer. Sending patches directly to Linus is not253normally the right way to go.254 255 256Next trees257----------258 259The chain of subsystem trees guides the flow of patches into the kernel,260but it also raises an interesting question: what if somebody wants to look261at all of the patches which are being prepared for the next merge window?262Developers will be interested in what other changes are pending to see263whether there are any conflicts to worry about; a patch which changes a264core kernel function prototype, for example, will conflict with any other265patches which use the older form of that function. Reviewers and testers266want access to the changes in their integrated form before all of those267changes land in the mainline kernel. One could pull changes from all of268the interesting subsystem trees, but that would be a big and error-prone269job.270 271The answer comes in the form of -next trees, where subsystem trees are272collected for testing and review. The older of these trees, maintained by273Andrew Morton, is called "-mm" (for memory management, which is how it got274started). The -mm tree integrates patches from a long list of subsystem275trees; it also has some patches aimed at helping with debugging.276 277Beyond that, -mm contains a significant collection of patches which have278been selected by Andrew directly. These patches may have been posted on a279mailing list, or they may apply to a part of the kernel for which there is280no designated subsystem tree. As a result, -mm operates as a sort of281subsystem tree of last resort; if there is no other obvious path for a282patch into the mainline, it is likely to end up in -mm. Miscellaneous283patches which accumulate in -mm will eventually either be forwarded on to284an appropriate subsystem tree or be sent directly to Linus. In a typical285development cycle, approximately 5-10% of the patches going into the286mainline get there via -mm.287 288The current -mm patch is available in the "mmotm" (-mm of the moment)289directory at:290 291 https://www.ozlabs.org/~akpm/mmotm/292 293Use of the MMOTM tree is likely to be a frustrating experience, though;294there is a definite chance that it will not even compile.295 296The primary tree for next-cycle patch merging is linux-next, maintained by297Stephen Rothwell. The linux-next tree is, by design, a snapshot of what298the mainline is expected to look like after the next merge window closes.299Linux-next trees are announced on the linux-kernel and linux-next mailing300lists when they are assembled; they can be downloaded from:301 302 https://www.kernel.org/pub/linux/kernel/next/303 304Linux-next has become an integral part of the kernel development process;305all patches merged during a given merge window should really have found306their way into linux-next some time before the merge window opens.307 308 309Staging trees310-------------311 312The kernel source tree contains the drivers/staging/ directory, where313many sub-directories for drivers or filesystems that are on their way to314being added to the kernel tree live. They remain in drivers/staging while315they still need more work; once complete, they can be moved into the316kernel proper. This is a way to keep track of drivers that aren't317up to Linux kernel coding or quality standards, but people may want to use318them and track development.319 320Greg Kroah-Hartman currently maintains the staging tree. Drivers that321still need work are sent to him, with each driver having its own322subdirectory in drivers/staging/. Along with the driver source files, a323TODO file should be present in the directory as well. The TODO file lists324the pending work that the driver needs for acceptance into the kernel325proper, as well as a list of people that should be Cc'd for any patches to326the driver. Current rules require that drivers contributed to staging327must, at a minimum, compile properly.328 329Staging can be a relatively easy way to get new drivers into the mainline330where, with luck, they will come to the attention of other developers and331improve quickly. Entry into staging is not the end of the story, though;332code in staging which is not seeing regular progress will eventually be333removed. Distributors also tend to be relatively reluctant to enable334staging drivers. So staging is, at best, a stop on the way toward becoming335a proper mainline driver.336 337 338Tools339-----340 341As can be seen from the above text, the kernel development process depends342heavily on the ability to herd collections of patches in various343directions. The whole thing would not work anywhere near as well as it344does without suitably powerful tools. Tutorials on how to use these tools345are well beyond the scope of this document, but there is space for a few346pointers.347 348By far the dominant source code management system used by the kernel349community is git. Git is one of a number of distributed version control350systems being developed in the free software community. It is well tuned351for kernel development, in that it performs quite well when dealing with352large repositories and large numbers of patches. It also has a reputation353for being difficult to learn and use, though it has gotten better over354time. Some sort of familiarity with git is almost a requirement for kernel355developers; even if they do not use it for their own work, they'll need git356to keep up with what other developers (and the mainline) are doing.357 358Git is now packaged by almost all Linux distributions. There is a home359page at:360 361 https://git-scm.com/362 363That page has pointers to documentation and tutorials.364 365Among the kernel developers who do not use git, the most popular choice is366almost certainly Mercurial:367 368 https://www.selenic.com/mercurial/369 370Mercurial shares many features with git, but it provides an interface which371many find easier to use.372 373The other tool worth knowing about is Quilt:374 375 https://savannah.nongnu.org/projects/quilt/376 377Quilt is a patch management system, rather than a source code management378system. It does not track history over time; it is, instead, oriented379toward tracking a specific set of changes against an evolving code base.380Some major subsystem maintainers use quilt to manage patches intended to go381upstream. For the management of certain kinds of trees (-mm, for example),382quilt is the best tool for the job.383 384 385Mailing lists386-------------387 388A great deal of Linux kernel development work is done by way of mailing389lists. It is hard to be a fully-functioning member of the community390without joining at least one list somewhere. But Linux mailing lists also391represent a potential hazard to developers, who risk getting buried under a392load of electronic mail, running afoul of the conventions used on the Linux393lists, or both.394 395Most kernel mailing lists are hosted at kernel.org; the master list can396be found at:397 398 https://subspace.kernel.org399 400There are lists hosted elsewhere; please check the MAINTAINERS file for401the list relevant for any particular subsystem.402 403The core mailing list for kernel development is, of course, linux-kernel.404This list is an intimidating place to be; volume can reach 500 messages per405day, the amount of noise is high, the conversation can be severely406technical, and participants are not always concerned with showing a high407degree of politeness. But there is no other place where the kernel408development community comes together as a whole; developers who avoid this409list will miss important information.410 411There are a few hints which can help with linux-kernel survival:412 413- Have the list delivered to a separate folder, rather than your main414 mailbox. One must be able to ignore the stream for sustained periods of415 time.416 417- Do not try to follow every conversation - nobody else does. It is418 important to filter on both the topic of interest (though note that419 long-running conversations can drift away from the original subject420 without changing the email subject line) and the people who are421 participating.422 423- Do not feed the trolls. If somebody is trying to stir up an angry424 response, ignore them.425 426- When responding to linux-kernel email (or that on other lists) preserve427 the Cc: header for all involved. In the absence of a strong reason (such428 as an explicit request), you should never remove recipients. Always make429 sure that the person you are responding to is in the Cc: list. This430 convention also makes it unnecessary to explicitly ask to be copied on431 replies to your postings.432 433- Search the list archives (and the net as a whole) before asking434 questions. Some developers can get impatient with people who clearly435 have not done their homework.436 437- Use interleaved ("inline") replies, which makes your response easier to438 read. (i.e. avoid top-posting -- the practice of putting your answer above439 the quoted text you are responding to.) For more details, see440 :ref:`Documentation/process/submitting-patches.rst <interleaved_replies>`.441 442- Ask on the correct mailing list. Linux-kernel may be the general meeting443 point, but it is not the best place to find developers from all444 subsystems.445 446The last point - finding the correct mailing list - is a common place for447beginning developers to go wrong. Somebody who asks a networking-related448question on linux-kernel will almost certainly receive a polite suggestion449to ask on the netdev list instead, as that is the list frequented by most450networking developers. Other lists exist for the SCSI, video4linux, IDE,451filesystem, etc. subsystems. The best place to look for mailing lists is452in the MAINTAINERS file packaged with the kernel source.453 454 455Getting started with Kernel development456---------------------------------------457 458Questions about how to get started with the kernel development process are459common - from both individuals and companies. Equally common are missteps460which make the beginning of the relationship harder than it has to be.461 462Companies often look to hire well-known developers to get a development463group started. This can, in fact, be an effective technique. But it also464tends to be expensive and does not do much to grow the pool of experienced465kernel developers. It is possible to bring in-house developers up to speed466on Linux kernel development, given the investment of a bit of time. Taking467this time can endow an employer with a group of developers who understand468the kernel and the company both, and who can help to train others as well.469Over the medium term, this is often the more profitable approach.470 471Individual developers are often, understandably, at a loss for a place to472start. Beginning with a large project can be intimidating; one often wants473to test the waters with something smaller first. This is the point where474some developers jump into the creation of patches fixing spelling errors or475minor coding style issues. Unfortunately, such patches create a level of476noise which is distracting for the development community as a whole, so,477increasingly, they are looked down upon. New developers wishing to478introduce themselves to the community will not get the sort of reception479they wish for by these means.480 481Andrew Morton gives this advice for aspiring kernel developers482 483::484 485 The #1 project for all kernel beginners should surely be "make sure486 that the kernel runs perfectly at all times on all machines which487 you can lay your hands on". Usually the way to do this is to work488 with others on getting things fixed up (this can require489 persistence!) but that's fine - it's a part of kernel development.490 491(https://lwn.net/Articles/283982/).492 493In the absence of obvious problems to fix, developers are advised to look494at the current lists of regressions and open bugs in general. There is495never any shortage of issues in need of fixing; by addressing these issues,496developers will gain experience with the process while, at the same time,497building respect with the rest of the development community.498