286 lines · plain
1================================2Fuzzing LLVM libraries and tools3================================4 5.. contents::6 :local:7 :depth: 28 9Introduction10============11 12The LLVM tree includes a number of fuzzers for various components. These are13built on top of :doc:`LibFuzzer <LibFuzzer>`. In order to build and run these14fuzzers, see :ref:`building-fuzzers`.15 16 17Available Fuzzers18=================19 20clang-fuzzer21------------22 23A |generic fuzzer| that tries to compile textual input as C++ code. Some of the24bugs this fuzzer has reported are `on bugzilla`__ and `on OSS Fuzz's25tracker`__.26 27__ https://llvm.org/pr2305728__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+clang-fuzzer29 30clang-proto-fuzzer31------------------32 33A |protobuf fuzzer| that compiles valid C++ programs generated from a protobuf34class that describes a subset of the C++ language.35 36This fuzzer accepts clang command-line options after `ignore_remaining_args=1`.37For example, the following command will fuzz clang with a higher optimization38level:39 40.. code-block:: shell41 42 % bin/clang-proto-fuzzer <corpus-dir> -ignore_remaining_args=1 -O343 44clang-format-fuzzer45-------------------46 47A |generic fuzzer| that runs clang-format_ on C++ text fragments. Some of the48bugs this fuzzer has reported are `on bugzilla`__49and `on OSS Fuzz's tracker`__.50 51.. _clang-format: https://clang.llvm.org/docs/ClangFormat.html52__ https://llvm.org/pr2305253__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+clang-format-fuzzer54 55llvm-as-fuzzer56--------------57 58A |generic fuzzer| that tries to parse text as :doc:`LLVM assembly <LangRef>`.59Some of the bugs this fuzzer has reported are `on bugzilla`__.60 61__ https://llvm.org/pr2463962 63llvm-dwarfdump-fuzzer64---------------------65 66A |generic fuzzer| that interprets inputs as object files and runs67:doc:`llvm-dwarfdump <CommandGuide/llvm-dwarfdump>` on them. Some of the bugs68this fuzzer has reported are `on OSS Fuzz's tracker`__69 70__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+llvm-dwarfdump-fuzzer71 72llvm-demangle-fuzzer73---------------------74 75A |generic fuzzer| for the Itanium demangler used in various LLVM tools. We've76fuzzed __cxa_demangle to death, why not fuzz LLVM's implementation of the same77function!78 79llvm-isel-fuzzer80----------------81 82A |LLVM IR fuzzer| aimed at finding bugs in instruction selection.83 84This fuzzer accepts flags after `ignore_remaining_args=1`. The flags match85those of :doc:`llc <CommandGuide/llc>` and the triple is required. For example,86the following command would fuzz AArch64 with :doc:`GlobalISel/index`:87 88.. code-block:: shell89 90 % bin/llvm-isel-fuzzer <corpus-dir> -ignore_remaining_args=1 -mtriple aarch64 -global-isel -O091 92Some flags can also be specified in the binary name itself in order to support93OSS Fuzz, which has trouble with required arguments. To do this, you can copy94or move ``llvm-isel-fuzzer`` to ``llvm-isel-fuzzer--x-y-z``, separating options95from the binary name using "--". The valid options are architecture names96(``aarch64``, ``x86_64``), optimization levels (``O0``, ``O2``), or specific97keywords, like ``gisel`` for enabling global instruction selection. In this98mode, the same example could be run like so:99 100.. code-block:: shell101 102 % bin/llvm-isel-fuzzer--aarch64-O0-gisel <corpus-dir>103 104llvm-opt-fuzzer105---------------106 107A |LLVM IR fuzzer| aimed at finding bugs in optimization passes.108 109It receives an optimization pipeline and runs it for each fuzzer input.110 111Interface of this fuzzer almost directly mirrors ``llvm-isel-fuzzer``. Both112``mtriple`` and ``passes`` arguments are required. Passes are specified in a113format suitable for the new pass manager. You can find some documentation about114this format in the doxygen for ``PassBuilder::parsePassPipeline``.115 116.. code-block:: shell117 118 % bin/llvm-opt-fuzzer <corpus-dir> -ignore_remaining_args=1 -mtriple x86_64 -passes instcombine119 120Similarly to the ``llvm-isel-fuzzer``, arguments in some predefined configurations121might be embedded directly into the binary file name:122 123.. code-block:: shell124 125 % bin/llvm-opt-fuzzer--x86_64-instcombine <corpus-dir>126 127llvm-mc-assemble-fuzzer128-----------------------129 130A |generic fuzzer| that fuzzes the MC layer's assemblers by treating inputs as131target-specific assembly.132 133Note that this fuzzer has an unusual command line interface which is not fully134compatible with all of libFuzzer's features. Fuzzer arguments must be passed135after ``--fuzzer-args``, and any ``llc`` flags must use two dashes. For136example, to fuzz the AArch64 assembler you might use the following command:137 138.. code-block:: console139 140 llvm-mc-fuzzer --triple=aarch64-linux-gnu --fuzzer-args -max_len=4141 142This scheme will likely change in the future.143 144llvm-mc-disassemble-fuzzer145--------------------------146 147A |generic fuzzer| that fuzzes the MC layer's disassemblers by treating inputs148as assembled binary data.149 150Note that this fuzzer has an unusual command line interface which is not fully151compatible with all of libFuzzer's features. See the notes above about152``llvm-mc-assemble-fuzzer`` for details.153 154 155.. |generic fuzzer| replace:: :ref:`generic fuzzer <fuzzing-llvm-generic>`156.. |protobuf fuzzer|157 replace:: :ref:`libprotobuf-mutator based fuzzer <fuzzing-llvm-protobuf>`158.. |LLVM IR fuzzer|159 replace:: :ref:`structured LLVM IR fuzzer <fuzzing-llvm-ir>`160 161lldb-target-fuzzer162---------------------163 164A |generic fuzzer| that interprets inputs as object files and uses them to165create a target in lldb.166 167Mutators and Input Generators168=============================169 170The inputs for a fuzz target are generated via random mutations of a171:ref:`corpus <libfuzzer-corpus>`. There are a few options for the kinds of172mutations that a fuzzer in LLVM might want.173 174.. _fuzzing-llvm-generic:175 176Generic Random Fuzzing177----------------------178 179The most basic form of input mutation is to use the built-in mutators of180LibFuzzer. These simply treat the input corpus as a bag of bits and make random181mutations. This type of fuzzer is good for stressing the surface layers of a182program, and is good at testing things like lexers, parsers, or binary183protocols.184 185Some of the in-tree fuzzers that use this type of mutator are `clang-fuzzer`_,186`clang-format-fuzzer`_, `llvm-as-fuzzer`_, `llvm-dwarfdump-fuzzer`_,187`llvm-mc-assemble-fuzzer`_, and `llvm-mc-disassemble-fuzzer`_.188 189.. _fuzzing-llvm-protobuf:190 191Structured Fuzzing using ``libprotobuf-mutator``192------------------------------------------------193 194We can use libprotobuf-mutator_ in order to perform structured fuzzing and195stress deeper layers of programs. This works by defining a protobuf class that196translates arbitrary data into structurally interesting input. Specifically, we197use this to work with a subset of the C++ language and perform mutations that198produce valid C++ programs in order to exercise parts of clang that are more199interesting than parser error handling.200 201To build this kind of fuzzer you need `protobuf`_ and its dependencies202installed, and you need to specify some extra flags when configuring the build203with :doc:`CMake <CMake>`. For example, `clang-proto-fuzzer`_ can be enabled by204adding ``-DCLANG_ENABLE_PROTO_FUZZER=ON`` to the flags described in205:ref:`building-fuzzers`.206 207The only in-tree fuzzer that uses ``libprotobuf-mutator`` today is208`clang-proto-fuzzer`_.209 210.. _libprotobuf-mutator: https://github.com/google/libprotobuf-mutator211.. _protobuf: https://github.com/google/protobuf212 213.. _fuzzing-llvm-ir:214 215Structured Fuzzing of LLVM IR216-----------------------------217 218We also use a more direct form of structured fuzzing for fuzzers that take219:doc:`LLVM IR <LangRef>` as input. This is achieved through the ``FuzzMutate``220library, which was `discussed at EuroLLVM 2017`_.221 222The ``FuzzMutate`` library is used to structurally fuzz backends in223`llvm-isel-fuzzer`_.224 225.. _discussed at EuroLLVM 2017: https://www.youtube.com/watch?v=UBbQ_s6hNgg226 227 228Building and Running229====================230 231.. _building-fuzzers:232 233Configuring LLVM to Build Fuzzers234---------------------------------235 236Fuzzers will be built and linked to libFuzzer by default as long as you build237LLVM with sanitizer coverage enabled. You would typically also enable at least238one sanitizer to find bugs faster. The most common way to build the fuzzers is239by adding the following two flags to your CMake invocation:240``-DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=On``.241 242.. note:: If you have ``compiler-rt`` checked out in an LLVM tree when building243 with sanitizers, you'll want to specify ``-DLLVM_BUILD_RUNTIME=Off``244 to avoid building the sanitizers themselves with sanitizers enabled.245 246.. note:: You may run into issues if you build with BFD ld, which is the247 default linker on many Unix systems. These issues are being tracked248 in https://llvm.org/PR34636.249 250Continuously Running and Finding Bugs251-------------------------------------252 253There used to be a public buildbot running LLVM fuzzers continuously, and while254this did find issues, it didn't have a very good way to report problems in an255actionable way. Because of this, we're moving towards using `OSS Fuzz`_ more256instead.257 258You can browse the `LLVM project issue list`_ for the bugs found by259`LLVM on OSS Fuzz`_. These are also mailed to the `llvm-bugs mailing260list`_.261 262.. _OSS Fuzz: https://github.com/google/oss-fuzz263.. _LLVM project issue list:264 https://bugs.chromium.org/p/oss-fuzz/issues/list?q=Proj-llvm265.. _LLVM on OSS Fuzz:266 https://github.com/google/oss-fuzz/blob/master/projects/llvm267.. _llvm-bugs mailing list:268 http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs269 270 271Utilities for Writing Fuzzers272=============================273 274There are some utilities available for writing fuzzers in LLVM.275 276Some helpers for handling the command line interface are available in277``include/llvm/FuzzMutate/FuzzerCLI.h``, including functions to parse command278line options in a consistent way and to implement standalone main functions so279your fuzzer can be built and tested when not built against libFuzzer.280 281There is also some handling of the CMake config for fuzzers, where you should282use the ``add_llvm_fuzzer`` to set up fuzzer targets. This function works283similarly to functions such as ``add_llvm_tool``, but it takes care of linking284to LibFuzzer when appropriate and can be passed the ``DUMMY_MAIN`` argument to285enable standalone testing.286