417 lines · plain
1===================================================================2How To Add Your Build Configuration To LLVM Buildbot Infrastructure3===================================================================4 5Introduction6============7 8This document contains information about adding a build configuration and9buildbot worker to the LLVM Buildbot Infrastructure.10 11.. note:: The term "buildmaster" is used in this document to refer to the12 server that manages which builds are run and where. Though we would not13 normally choose to use "master" terminology, it is used in this document14 because it is the term that the Buildbot package currently15 `uses <https://github.com/buildbot/buildbot/issues/5382>`_.16 17Buildmasters18============19 20There are two buildmasters running.21 22* The main buildmaster at `<https://lab.llvm.org/buildbot>`_. All builders23 attached to this machine will notify commit authors every time they break24 the build.25* The staging buildmaster at `<https://lab.llvm.org/staging>`_. All builders26 attached to this machine will be completely silent by default when the build27 is broken. This buildmaster is reconfigured every two hours with any new28 commits from the llvm-zorg repository.29 30In order to remain connected to the main buildmaster (and thus notify31developers of failures), a buildbot must:32 33* Be building a supported configuration. Builders for experimental backends34 should generally be attached to staging buildmaster.35* Be able to keep up with new commits to the main branch, or at a minimum36 recover to tip of tree within a couple of days of falling behind.37 38Additionally, we encourage all bot owners to point their bots towards the39staging master during maintenance windows, instability troubleshooting, and40such.41 42Roles & Expectations43====================44 45Each buildbot has an owner who is the responsible party for addressing problems46which arise with said buildbot. We generally expect the bot owner to be47reasonably responsive.48 49For some bots, the ownership responsibility is split between a "resource owner"50who provides the underlying machine resource, and a "configuration owner" who51maintains the build configuration. Generally, operational responsibility lies52with the "config owner". We do expect "resource owners" - who are generally53the contact listed in a workers attributes - to proxy requests to the relevant54"config owner" in a timely manner.55 56Most issues with a buildbot should be addressed directly with a bot owner57via email. Please CC `Galina Kistanova <mailto:gkistanova@gmail.com>`_.58 59Steps To Add Builder To LLVM Buildbot60=====================================61Volunteers can provide their build machines to work as build workers to62public LLVM Buildbot.63 64Here are the steps you can follow to do so:65 66#. Check the existing build configurations to make sure the one you are67 interested in is not covered yet or gets built on your computer much68 faster than on the existing one. We prefer faster builds so developers69 will get feedback sooner after changes get committed.70 71#. The computer you will be registering with the LLVM buildbot72 infrastructure should have all dependencies installed and be able to73 build your configuration successfully. Please check what degree74 of parallelism (-j param) would give the fastest build. You can build75 multiple configurations on one computer.76 77#. Install buildbot-worker (currently we are using buildbot version 3.11.7).78 This specific version can be installed using ``pip``, with a command such79 as ``pip3 install buildbot-worker==3.11.7``.80 81#. Create a designated user account, your buildbot-worker will be running under,82 and set appropriate permissions.83 84#. Choose the buildbot-worker root directory (all builds will be placed under85 it), buildbot-worker access name and password the build master will be using86 to authenticate your buildbot-worker.87 88#. Create a buildbot-worker in context of that buildbot-worker account. Point it89 to the **lab.llvm.org** port **9994** (see `Buildbot documentation,90 Creating a worker91 <http://docs.buildbot.net/current/tutorial/firstrun.html#creating-a-worker>`_92 for more details) by running the following command:93 94 .. code-block:: bash95 96 $ buildbot-worker create-worker <buildbot-worker-root-directory> \97 lab.llvm.org:9994 \98 <buildbot-worker-access-name> \99 <buildbot-worker-access-password>100 101 Only once a new worker is stable, and102 approval from Galina has been received (see last step) should it103 be pointed at the main buildmaster.104 105 Now start the worker:106 107 .. code-block:: bash108 109 $ buildbot-worker start <buildbot-worker-root-directory>110 111 This will cause your new worker to connect to the staging buildmaster112 which is silent by default.113 114 Try this once then check the log file115 ``<buildbot-worker-root-directory>/worker/twistd.log``. If your settings116 are correct you will see a refused connection. This is good and expected,117 as the credentials have not been established on both ends. Now stop the118 worker and proceed to the next steps.119 120#. Fill the buildbot-worker description and admin name/e-mail. Here is an121 example of the buildbot-worker description::122 123 Windows 7 x64124 Core i7 (2.66GHz), 16GB of RAM125 126 g++.exe (TDM-1 mingw32) 4.4.0127 GNU Binutils 2.19.1128 cmake version 2.8.4129 Microsoft(R) 32-bit C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86130 131 See `here <http://docs.buildbot.net/current/manual/installation/worker.html>`_132 for which files to edit.133 134#. Send a patch which adds your build worker and your builder to135 `zorg <https://github.com/llvm/llvm-zorg>`_. Use the typical LLVM136 `workflow <https://llvm.org/docs/Contributing.html#how-to-submit-a-patch>`_.137 138 * workers are added to ``buildbot/osuosl/master/config/workers.py``139 * builders are added to ``buildbot/osuosl/master/config/builders.py``140 141 Please make sure your builder name and its builddir are unique through the142 file.143 144 All new builders should default to using the "'collapseRequests': False"145 configuration. This causes the builder to build each commit individually146 and not merge build requests. To maximize quality of feedback to developers,147 we *strongly prefer* builders to be configured not to collapse requests.148 This flag should be removed only after all reasonable efforts have been149 exhausted to improve build times such that the builder can keep up with150 commit flow.151 152 It is possible to allow email addresses to unconditionally receive153 notifications on build failure; for this you'll need to add an154 ``InformativeMailNotifier`` to ``buildbot/osuosl/master/config/status.py``.155 This is particularly useful for the staging buildmaster which is silent156 otherwise.157 158#. Send the buildbot-worker access name and the access password directly to159 `Galina Kistanova <mailto:gkistanova@gmail.com>`_, and wait until she160 lets you know that your changes are applied and buildmaster is161 reconfigured.162 163#. Make sure you can start the buildbot-worker and successfully connect164 to the silent buildmaster. Then set up your buildbot-worker to start165 automatically at the start up time. See the buildbot documentation166 for help. You may want to restart your computer to see if it works.167 168#. Check the status of your buildbot-worker on the `Waterfall Display (Staging)169 <http://lab.llvm.org/staging/#/waterfall>`_ to make sure it is170 connected, and the `Workers Display (Staging)171 <http://lab.llvm.org/staging/#/workers>`_ to see if administrator172 contact and worker information are correct.173 174#. At this point, you have a working builder connected to the staging175 buildmaster. You can now make sure it is reliably green and keeps176 up with the build queue. No notifications will be sent, so you can177 keep an unstable builder connected to staging indefinitely.178 179#. (Optional) Once the builder is stable on the staging buildmaster with180 several days of green history, you can choose to move it to the production181 buildmaster to enable developer notifications. Please email `Galina182 Kistanova <mailto:gkistanova@gmail.com>`_ for review and approval.183 184 To move a worker to production (once approved), stop your worker, edit the185 buildbot.tac file to change the port number from 9994 to 9990 and start it186 again.187 188Testing a Builder Config Locally189================================190 191It is possible to test a builder running against a local version of LLVM's192buildmaster setup. This allows you to test changes to builder, worker, and193buildmaster configuration. A buildmaster launched in this "local testing" mode194will:195 196* Bind only to local interfaces.197* Use SQLite as the database.198* Use a single fixed password for workers.199* Disable extras like GitHub authentication.200 201In order to use this "local testing" mode:202 203* Create and activate a Python `venv204 <https://docs.python.org/3/library/venv.html>`_ and install the necessary205 dependencies. This step can be run from any directory.206 207 .. code-block:: bash208 209 python -m venv bbenv210 source bbenv/bin/activate211 pip install buildbot{,-console-view,-grid-view,-waterfall-view,-worker,-www}==3.11.7 urllib3212 213* If your system has Python 3.13 or newer you will need to additionally214 install ``legacy-cgi`` and make a minor patch to the installed buildbot215 package. This step does not need to be followed for earlier Python versions.216 217 .. code-block:: bash218 219 pip install legacy-cgi220 sed -i \221 -e 's/import pipes/import shlex/' \222 -e 's/pipes\.quote/shlex.quote/' \223 bbenv/lib/python3.13/site-packages/buildbot_worker/runprocess.py224 225* Initialise the necessary buildmaster files, link to the configuration in a226 local checkout out of `llvm-zorg <https://github.com/llvm/llvm-zorg>`_, and227 ask ``buildbot`` to check the configuration. This step can be run from any228 directory.229 230 .. code-block:: bash231 232 buildbot create-master llvm-testbbmaster233 cd llvm-testbbmaster234 ln -s /path/to/checkout/of/llvm-zorg/buildbot/osuosl/master/master.cfg .235 ln -s /path/to/checkout/of/llvm-zorg/buildbot/osuosl/master/config/ .236 ln -s /path/to/checkout/of/llvm-zorg/zorg/ .237 BUILDBOT_TEST=1 buildbot checkconfig238 239* Start the buildmaster.240 241 .. code-block:: bash242 243 BUILDBOT_TEST=1 buildbot start --nodaemon .244 245* After waiting a few seconds for startup to complete, you should be able to246 open the web UI at ``http://localhost:8011``. If there are any errors or247 this isn't working, check ``twistd.log`` (within the current directory) for248 more information.249 250* You can now create and start a buildbot worker. Ensure you pick the correct251 name for the worker associated with the build configuration you want to test252 in ``buildbot/osuosl/master/config/builders.py``.253 254 .. code-block:: bash255 256 buildbot-worker create-worker <buildbot-worker-root-directory> \257 localhost:9990 \258 <buildbot-worker-name> \259 test260 buildbot-worker start --nodaemon <buildbot-worker-root-directory>261 262* Either wait until the poller sets off a build, or alternatively force a263 build to start in the web UI.264 265* Review the progress and results of the build in the web UI.266 267This local testing configuration defaults to binding only to the loopback268interface for security reasons.269 270If you want to run the test worker on a different machine, or to run the271buildmaster on a remote server, ssh port forwarding can be used to make272connection possible. For instance, if running the buildmaster on a remote273server the following command will suffice to make the web UI accessible via274``http://localhost:8011`` and make it possible for a local worker to connect275to the remote buildmaster by connecting to ``localhost:9900``:276 277 .. code-block:: bash278 279 ssh -N -L 8011:localhost:8011 -L 9990:localhost:9990 username@buildmaster_server_address280 281Be aware that some build configurations may checkout the current upstream282``llvm-zorg`` repository in order to retrieve additional scripts used during283the build process, meaning any local changes will not be reflected in this284part of the build. If you wish to test changes to any of these scripts without285committing them upstream, you will need to temporarily patch the builder logic286in order to instead check out your own branch.287Typically, ``addGetSourcecodeForProject`` from288``zorg/buildbot/process/factory.py`` is used for this and you can edit the289caller to specify your own ``repourl`` and/or ``branch`` keyword argument.290 291Best Practices for Configuring a Fast Builder292=============================================293 294As mentioned above, we generally have a strong preference for295builders which can build every commit as they come in. This section296includes best practices and some recommendations as to how to achieve297that end.298 299The goal300 In 2020, the monorepo had just under 35 thousand commits. This works301 out to an average of 4 commits per hour. Already, we can see that a302 builder must cycle in less than 15 minutes to have a hope of being303 useful. However, those commits are not uniformly distributed. They304 tend to cluster strongly during US working hours. Looking at a couple305 of recent (Nov 2021) working days, we routinely see ~10 commits per306 hour during peek times, with occasional spikes as high as ~15 commits307 per hour. Thus, as a rule of thumb, we should plan for our builder to308 complete ~10-15 builds an hour.309 310Resource Appropriately311 At 10-15 builds per hour, we need to complete a new build on average every312 4 to 6 minutes. For anything except the fastest of hardware/build configs,313 this is going to be well beyond the ability of a single machine. In buildbot314 terms, we likely going to need multiple workers to build requests in parallel315 under a single builder configuration. For some rough back of the envelope316 numbers, if your build config takes e.g. 30 minutes, you will need something317 on the order of 5-8 workers. If your build config takes ~2 hours, you'll318 need something on the order of 20-30 workers. The rest of this section319 focuses on how to reduce cycle times.320 321Restrict what you build and test322 Think hard about why you're setting up a bot, and restrict your build323 configuration as much as you can. Basic functionality is probably324 already covered by other bots, and you don't need to duplicate that325 testing. You only need to be building and testing the *unique* parts326 of the configuration. (e.g. For a multi-stage clang builder, you probably327 don't need to be enabling every target or building all the various utilities.)328 329 It can sometimes be worthwhile splitting a single builder into two or more,330 if you have multiple distinct purposes for the same builder. As an example,331 if you want to both a) confirm that all of LLVM builds with your host332 compiler, and b) want to do a multi-stage clang build on your target, you333 may be better off with two separate bots. Splitting increases resource334 consumption, but makes it easy for each bot to keep up with commit flow.335 Additionally, splitting bots may assist in triage by narrowing attention to336 relevant parts of the failing configuration.337 338 In general, we recommend Release build types with Assertions enabled. This339 generally provides a good balance between build times and bug detection for340 most buildbots. There may be room for including some debug info (e.g. with341 `-gmlt`), but in general the balance between debug info quality and build342 times is a delicate one.343 344Use Ninja & LLD345 Ninja really does help build times over Make, particularly for highly346 parallel builds. LLD helps to reduce both link times and memory usage347 during linking significantly. With a build machine with sufficient348 parallelism, link times tend to dominate critical path of the build, and are349 thus worth optimizing.350 351Use CCache and NOT incremental builds352 Using ccache materially improves average build times. Incremental builds353 can be slightly faster, but introduce the risk of build corruption due to354 e.g. state changes, etc... At this point, the recommendation is not to355 use incremental builds and instead use ccache as the latter captures the356 majority of the benefit with less risk of false positives.357 358 One of the non-obvious benefits of using ccache is that it makes the359 builder less sensitive to which projects are being monitored vs built.360 If a change triggers a build request, but doesn't change the build output361 (e.g. doc changes, python utility changes, etc..), the build will entirely362 hit in cache and the build request will complete in just the testing time.363 364 With multiple workers, it is tempting to try to configure a shared cache365 between the workers. Experience to date indicates this is difficult to366 well, and that having local per-worker caches gets most of the benefit367 anyways. We don't currently recommend shared caches.368 369 CCache does depend on the builder hardware having sufficient IO to access370 the cache with reasonable access times - i.e. a fast disk, or enough memory371 for a RAM cache, etc.. For builders without, incremental may be your best372 option, but is likely to require higher ongoing involvement from the373 sponsor.374 375Enable batch builds376 As a last resort, you can configure your builder to batch build requests.377 This makes the build failure notifications markedly less actionable, and378 should only be done once all other reasonable measures have been taken.379 380Leave it on the staging buildmaster381 While most of this section has been biased towards builders intended for382 the main buildmaster, it is worth highlighting that builders can run383 indefinitely on the staging buildmaster. Such a builder may still be384 useful for the sponsoring organization, without concern of negatively385 impacting the broader community. The sponsoring organization simply386 has to take on the responsibility of all bisection and triage.387 388Managing a Worker From The Web Interface389========================================390 391Tasks such as clearing pending building requests can be done using392the Buildbot web interface. To do this you must be recognised as an admin393of the worker:394 395* Set your public GitHub profile email to one that was included in the396 ``admin`` information you set up on the worker. It does not matter if this397 is your primary account email or a "verified email". To confirm this has been398 done correctly, go to ``github.com/<your GitHub username>`` and you should399 see the email address listed there.400 401 A worker can have many admins, if they are listed in the form402 ``First Last <first.last@example.com>, First2 Last2 <first2.last2@example.com>``.403 You only need to have one of those addresses in your profile to be recognised404 as an admin.405 406 If you need to add an email address, you can edit the ``admin`` file and407 restart the worker. You should see the new admin details in the web interface408 shortly afterwards.409 410* Connect GitHub to Buildbot by clicking on the "Anonymous" button on the411 top right of the page, then "Login with GitHub" and authorise the app.412 413Some tasks don't give immediate feedback, so if nothing happens within a short414time, try again with the browser's web console open. Sometimes you will see415403 errors and other messages that might indicate you don't have the correct416details set up.417