88 lines · plain
1==================================2Benchmarking tips3==================================4 5 6Introduction7============8 9For benchmarking a patch we want to reduce all possible sources of10noise as much as possible. How to do that is very OS dependent.11 12Note that low noise is required, but not sufficient. It does not13exclude measurement bias.14See `"Producing Wrong Data Without Doing Anything Obviously Wrong!" by Mytkowicz, Diwan, Hauswith and Sweeney (ASPLOS 2009) <https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf>`_15for example.16 17General18================================19 20* Use a high-resolution timer, e.g., perf under Linux.21 22* Run the benchmark multiple times to be able to recognize noise.23 24* Disable as many processes or services as possible on the target system.25 26* Disable frequency scaling, Turbo Boost and address space27 randomization (see OS-specific section).28 29* Use static linking if the OS supports it. That avoids any variation that30 might be introduced by loading dynamic libraries. This can be done31 by passing ``-DLLVM_BUILD_STATIC=ON`` to CMake.32 33* Try to avoid storage. On some systems, you can use tmpfs. Putting the34 program, inputs and outputs on tmpfs avoids touching a real storage35 system, which can have a pretty big variability.36 37 To mount it (on Linux and FreeBSD at least)::38 39 mount -t tmpfs -o size=<XX>g none dir_to_mount40 41Linux42=====43 44* Disable address space randomization::45 46 echo 0 > /proc/sys/kernel/randomize_va_space47 48* Set scaling_governor to performance::49 50 for i in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor51 do52 echo performance > $i53 done54 55* Use https://github.com/lpechacek/cpuset to reserve CPU cores for just the56 program you are benchmarking. If using perf, leave at least 2 cores57 so that perf runs in one and your program in another::58 59 cset shield -c N1,N2 -k on60 61 This will move all threads out of N1 and N2. The ``-k on`` means62 that even kernel threads are moved out.63 64* Disable the SMT pair of the cpus you will use for the benchmark. The65 pair of cpu N can be found in66 ``/sys/devices/system/cpu/cpuN/topology/thread_siblings_list`` and67 disabled with::68 69 echo 0 > /sys/devices/system/cpu/cpuX/online70 71 72* Run the program with::73 74 cset shield --exec -- perf stat -r 10 <cmd>75 76 This will run the command after ``--`` in the isolated CPU cores. The77 particular perf command runs the ``<cmd>`` 10 times and reports78 statistics.79 80With these in place you can expect perf variations of less than 0.1%.81 82Linux Intel83-----------84 85* Disable Turbo Boost::86 87 echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo88