520 lines · plain
1.. Copyright 2010 Nicolas Palix <npalix@diku.dk>2.. Copyright 2010 Julia Lawall <julia@diku.dk>3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr>4 5.. highlight:: none6 7.. _devtools_coccinelle:8 9Coccinelle10==========11 12Coccinelle is a tool for pattern matching and text transformation that has13many uses in kernel development, including the application of complex,14tree-wide patches and detection of problematic programming patterns.15 16Getting Coccinelle17------------------18 19The semantic patches included in the kernel use features and options20which are provided by Coccinelle version 1.0.0-rc11 and above.21Using earlier versions will fail as the option names used by22the Coccinelle files and coccicheck have been updated.23 24Coccinelle is available through the package manager25of many distributions, e.g. :26 27 - Debian28 - Fedora29 - Ubuntu30 - OpenSUSE31 - Arch Linux32 - NetBSD33 - FreeBSD34 35Some distribution packages are obsolete and it is recommended36to use the latest version released from the Coccinelle homepage at37http://coccinelle.lip6.fr/38 39Or from Github at:40 41https://github.com/coccinelle/coccinelle42 43Once you have it, run the following commands::44 45 ./autogen46 ./configure47 make48 49as a regular user, and install it with::50 51 sudo make install52 53More detailed installation instructions to build from source can be54found at:55 56https://github.com/coccinelle/coccinelle/blob/master/install.txt57 58Supplemental documentation59--------------------------60 61For supplemental documentation refer to the wiki:62 63https://bottest.wiki.kernel.org/coccicheck64 65The wiki documentation always refers to the linux-next version of the script.66 67For Semantic Patch Language(SmPL) grammar documentation refer to:68 69https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html70 71Using Coccinelle on the Linux kernel72------------------------------------73 74A Coccinelle-specific target is defined in the top level75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck``76front-end in the ``scripts`` directory.77 78Four basic modes are defined: ``patch``, ``report``, ``context``, and79``org``. The mode to use is specified by setting the MODE variable with80``MODE=<mode>``.81 82- ``patch`` proposes a fix, when possible.83 84- ``report`` generates a list in the following format:85 file:line:column-column: message86 87- ``context`` highlights lines of interest and their context in a88 diff-like style. Lines of interest are indicated with ``-``.89 90- ``org`` generates a report in the Org mode format of Emacs.91 92Note that not all semantic patches implement all modes. For easy use93of Coccinelle, the default mode is "report".94 95Two other modes provide some common combinations of these modes.96 97- ``chain`` tries the previous modes in the order above until one succeeds.98 99- ``rep+ctxt`` runs successively the report mode and the context mode.100 It should be used with the C option (described later)101 which checks the code on a file basis.102 103Examples104~~~~~~~~105 106To make a report for every semantic patch, run the following command::107 108 make coccicheck MODE=report109 110To produce patches, run::111 112 make coccicheck MODE=patch113 114 115The coccicheck target applies every semantic patch available in the116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel.117 118For each semantic patch, a commit message is proposed. It gives a119description of the problem being checked by the semantic patch, and120includes a reference to Coccinelle.121 122As with any static code analyzer, Coccinelle produces false123positives. Thus, reports must be carefully checked, and patches124reviewed.125 126To enable verbose messages set the V= variable, for example::127 128 make coccicheck MODE=report V=1129 130Coccinelle parallelization131--------------------------132 133By default, coccicheck tries to run as parallel as possible. To change134the parallelism, set the J= variable. For example, to run across 4 CPUs::135 136 make coccicheck MODE=report J=4137 138As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization;139if support for this is detected you will benefit from parmap parallelization.140 141When parmap is enabled coccicheck will enable dynamic load balancing by using142``--chunksize 1`` argument. This ensures we keep feeding threads with work143one by one, so that we avoid the situation where most work gets done by only144a few threads. With dynamic load balancing, if a thread finishes early we keep145feeding it more work.146 147When parmap is enabled, if an error occurs in Coccinelle, this error148value is propagated back, and the return value of the ``make coccicheck``149command captures this return value.150 151Using Coccinelle with a single semantic patch152---------------------------------------------153 154The optional make variable COCCI can be used to check a single155semantic patch. In that case, the variable must be initialized with156the name of the semantic patch to apply.157 158For instance::159 160 make coccicheck COCCI=<my_SP.cocci> MODE=patch161 162or::163 164 make coccicheck COCCI=<my_SP.cocci> MODE=report165 166 167Controlling Which Files are Processed by Coccinelle168---------------------------------------------------169 170By default the entire kernel source tree is checked.171 172To apply Coccinelle to a specific directory, ``M=`` can be used.173For example, to check drivers/net/wireless/ one may write::174 175 make coccicheck M=drivers/net/wireless/176 177To apply Coccinelle on a file basis, instead of a directory basis, the178C variable is used by the makefile to select which files to work with.179This variable can be used to run scripts for the entire kernel, a180specific directory, or for a single file.181 182For example, to check drivers/bluetooth/bfusb.c, the value 1 is183passed to the C variable to check files that make considers184need to be compiled.::185 186 make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o187 188The value 2 is passed to the C variable to check files regardless of189whether they need to be compiled or not.::190 191 make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o192 193In these modes, which work on a file basis, there is no information194about semantic patches displayed, and no commit message proposed.195 196This runs every semantic patch in scripts/coccinelle by default. The197COCCI variable may additionally be used to only apply a single198semantic patch as shown in the previous section.199 200The "report" mode is the default. You can select another one with the201MODE variable explained above.202 203Debugging Coccinelle SmPL patches204---------------------------------205 206Using coccicheck is best as it provides in the spatch command line207include options matching the options used when we compile the kernel.208You can learn what these options are by using V=1; you could then209manually run Coccinelle with debug options added.210 211Alternatively you can debug running Coccinelle against SmPL patches212by asking for stderr to be redirected to stderr. By default stderr213is redirected to /dev/null; if you'd like to capture stderr you214can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For215instance::216 217 rm -f cocci.err218 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err219 cat cocci.err220 221You can use SPFLAGS to add debugging flags; for instance you may want to222add both ``--profile --show-trying`` to SPFLAGS when debugging. For example223you may want to use::224 225 rm -f err.log226 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci227 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd228 229err.log will now have the profiling information, while stdout will230provide some progress information as Coccinelle moves forward with231work.232 233NOTE:234 235DEBUG_FILE support is only supported when using coccinelle >= 1.0.2.236 237Currently, DEBUG_FILE support is only available to check folders, and238not single files. This is because checking a single file requires spatch239to be called twice leading to DEBUG_FILE being set both times to the same value,240giving rise to an error.241 242.cocciconfig support243--------------------244 245Coccinelle supports reading .cocciconfig for default Coccinelle options that246should be used every time spatch is spawned. The order of precedence for247variables for .cocciconfig is as follows:248 249- Your current user's home directory is processed first250- Your directory from which spatch is called is processed next251- The directory provided with the ``--dir`` option is processed last, if used252 253Since coccicheck runs through make, it naturally runs from the kernel254proper dir; as such the second rule above would be implied for picking up a255.cocciconfig when using ``make coccicheck``.256 257``make coccicheck`` also supports using M= targets. If you do not supply258any M= target, it is assumed you want to target the entire kernel.259The kernel coccicheck script has::260 261 if [ "$KBUILD_EXTMOD" = "" ] ; then262 OPTIONS="--dir $srctree $COCCIINCLUDE"263 else264 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"265 fi266 267KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases268the spatch ``--dir`` argument is used, as such third rule applies when whether269M= is used or not, and when M= is used the target directory can have its own270.cocciconfig file. When M= is not passed as an argument to coccicheck the271target directory is the same as the directory from where spatch was called.272 273If not using the kernel's coccicheck target, keep the above precedence274order logic of .cocciconfig reading. If using the kernel's coccicheck target,275override any of the kernel's .coccicheck's settings using SPFLAGS.276 277We help Coccinelle when used against Linux with a set of sensible default278options for Linux with our own Linux .cocciconfig. This hints to coccinelle279that git can be used for ``git grep`` queries over coccigrep. A timeout of 200280seconds should suffice for now.281 282The options picked up by coccinelle when reading a .cocciconfig do not appear283as arguments to spatch processes running on your system. To confirm what284options will be used by Coccinelle run::285 286 spatch --print-options-only287 288You can override with your own preferred index option by using SPFLAGS. Take289note that when there are conflicting options Coccinelle takes precedence for290the last options passed. Using .cocciconfig is possible to use idutils, however291given the order of precedence followed by Coccinelle, since the kernel now292carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if293desired. See below section "Additional flags" for more details on how to use294idutils.295 296Additional flags297----------------298 299Additional flags can be passed to spatch through the SPFLAGS300variable. This works as Coccinelle respects the last flags301given to it when options are in conflict. ::302 303 make SPFLAGS=--use-glimpse coccicheck304 305Coccinelle supports idutils as well but requires coccinelle >= 1.0.6.306When no ID file is specified coccinelle assumes your ID database file307is in the file .id-utils.index on the top level of the kernel. Coccinelle308carries a script scripts/idutils_index.sh which creates the database with::309 310 mkid -i C --output .id-utils.index311 312If you have another database filename you can also just symlink with this313name. ::314 315 make SPFLAGS=--use-idutils coccicheck316 317Alternatively you can specify the database filename explicitly, for318instance::319 320 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck321 322See ``spatch --help`` to learn more about spatch options.323 324Note that the ``--use-glimpse`` and ``--use-idutils`` options325require external tools for indexing the code. None of them is326thus active by default. However, by indexing the code with327one of these tools, and according to the cocci file used,328spatch could proceed the entire code base more quickly.329 330SmPL patch specific options331---------------------------332 333SmPL patches can have their own requirements for options passed334to Coccinelle. SmPL patch-specific options can be provided by335providing them at the top of the SmPL patch, for instance::336 337 // Options: --no-includes --include-headers338 339SmPL patch Coccinelle requirements340----------------------------------341 342As Coccinelle features get added some more advanced SmPL patches343may require newer versions of Coccinelle. If an SmPL patch requires344a minimum version of Coccinelle, this can be specified as follows,345as an example if requiring at least Coccinelle >= 1.0.5::346 347 // Requires: 1.0.5348 349Proposing new semantic patches350------------------------------351 352New semantic patches can be proposed and submitted by kernel353developers. For sake of clarity, they should be organized in the354sub-directories of ``scripts/coccinelle/``.355 356 357Detailed description of the ``report`` mode358-------------------------------------------359 360``report`` generates a list in the following format::361 362 file:line:column-column: message363 364Example365~~~~~~~366 367Running::368 369 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci370 371will execute the following part of the SmPL script::372 373 <smpl>374 @r depends on !context && !patch && (org || report)@375 expression x;376 position p;377 @@378 379 ERR_PTR@p(PTR_ERR(x))380 381 @script:python depends on report@382 p << r.p;383 x << r.x;384 @@385 386 msg="ERR_CAST can be used with %s" % (x)387 coccilib.report.print_report(p[0], msg)388 </smpl>389 390This SmPL excerpt generates entries on the standard output, as391illustrated below::392 393 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg394 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth395 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg396 397 398Detailed description of the ``patch`` mode399------------------------------------------400 401When the ``patch`` mode is available, it proposes a fix for each problem402identified.403 404Example405~~~~~~~406 407Running::408 409 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci410 411will execute the following part of the SmPL script::412 413 <smpl>414 @ depends on !context && patch && !org && !report @415 expression x;416 @@417 418 - ERR_PTR(PTR_ERR(x))419 + ERR_CAST(x)420 </smpl>421 422This SmPL excerpt generates patch hunks on the standard output, as423illustrated below::424 425 diff -u -p a/crypto/ctr.c b/crypto/ctr.c426 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200427 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200428 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct429 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,430 CRYPTO_ALG_TYPE_MASK);431 if (IS_ERR(alg))432 - return ERR_PTR(PTR_ERR(alg));433 + return ERR_CAST(alg);434 435 /* Block size must be >= 4 bytes. */436 err = -EINVAL;437 438Detailed description of the ``context`` mode439--------------------------------------------440 441``context`` highlights lines of interest and their context442in a diff-like style.443 444 **NOTE**: The diff-like output generated is NOT an applicable patch. The445 intent of the ``context`` mode is to highlight the important lines446 (annotated with minus, ``-``) and gives some surrounding context447 lines around. This output can be used with the diff mode of448 Emacs to review the code.449 450Example451~~~~~~~452 453Running::454 455 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci456 457will execute the following part of the SmPL script::458 459 <smpl>460 @ depends on context && !patch && !org && !report@461 expression x;462 @@463 464 * ERR_PTR(PTR_ERR(x))465 </smpl>466 467This SmPL excerpt generates diff hunks on the standard output, as468illustrated below::469 470 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing471 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200472 +++ /tmp/nothing473 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct474 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,475 CRYPTO_ALG_TYPE_MASK);476 if (IS_ERR(alg))477 - return ERR_PTR(PTR_ERR(alg));478 479 /* Block size must be >= 4 bytes. */480 err = -EINVAL;481 482Detailed description of the ``org`` mode483----------------------------------------484 485``org`` generates a report in the Org mode format of Emacs.486 487Example488~~~~~~~489 490Running::491 492 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci493 494will execute the following part of the SmPL script::495 496 <smpl>497 @r depends on !context && !patch && (org || report)@498 expression x;499 position p;500 @@501 502 ERR_PTR@p(PTR_ERR(x))503 504 @script:python depends on org@505 p << r.p;506 x << r.x;507 @@508 509 msg="ERR_CAST can be used with %s" % (x)510 msg_safe=msg.replace("[","@(").replace("]",")")511 coccilib.org.print_todo(p[0], msg_safe)512 </smpl>513 514This SmPL excerpt generates Org entries on the standard output, as515illustrated below::516 517 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]]518 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]]519 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]]520