340 lines · plain
1==================2BPF Selftest Notes3==================4General instructions on running selftests can be found in5`Documentation/bpf/bpf_devel_QA.rst`__.6 7__ /Documentation/bpf/bpf_devel_QA.rst#q-how-to-run-bpf-selftests8 9=============10BPF CI System11=============12 13BPF employs a continuous integration (CI) system to check patch submission in an14automated fashion. The system runs selftests for each patch in a series. Results15are propagated to patchwork, where failures are highlighted similar to16violations of other checks (such as additional warnings being emitted or a17``scripts/checkpatch.pl`` reported deficiency):18 19 https://patchwork.kernel.org/project/netdevbpf/list/?delegate=12117320 21The CI system executes tests on multiple architectures. It uses a kernel22configuration derived from both the generic and architecture specific config23file fragments below ``tools/testing/selftests/bpf/`` (e.g., ``config`` and24``config.x86_64``).25 26Denylisting Tests27=================28 29It is possible for some architectures to not have support for all BPF features.30In such a case tests in CI may fail. An example of such a shortcoming is BPF31trampoline support on IBM's s390x architecture. For cases like this, an in-tree32deny list file, located at ``tools/testing/selftests/bpf/DENYLIST.<arch>``, can33be used to prevent the test from running on such an architecture.34 35In addition to that, the generic ``tools/testing/selftests/bpf/DENYLIST`` is36honored on every architecture running tests.37 38These files are organized in three columns. The first column lists the test in39question. This can be the name of a test suite or of an individual test. The40remaining two columns provide additional meta data that helps identify and41classify the entry: column two is a copy and paste of the error being reported42when running the test in the setting in question. The third column, if43available, summarizes the underlying problem. A value of ``trampoline``, for44example, indicates that lack of trampoline support is causing the test to fail.45This last entry helps identify tests that can be re-enabled once such support is46added.47 48=========================49Running Selftests in a VM50=========================51 52It's now possible to run the selftests using ``tools/testing/selftests/bpf/vmtest.sh``.53The script tries to ensure that the tests are run with the same environment as they54would be run post-submit in the CI used by the Maintainers, with the exception55that deny lists are not automatically honored.56 57This script uses the in-tree kernel configuration and downloads a VM userspace58image from the system used by the CI. It builds the kernel (without overwriting59your existing Kconfig), recompiles the bpf selftests, runs them (by default60``tools/testing/selftests/bpf/test_progs``) and saves the resulting output (by61default in ``~/.bpf_selftests``).62 63Script dependencies:64- clang (preferably built from sources, https://github.com/llvm/llvm-project);65- pahole (preferably built from sources, https://git.kernel.org/pub/scm/devel/pahole/pahole.git/);66- qemu;67- docutils (for ``rst2man``);68- libcap-devel.69 70For more information about using the script, run:71 72.. code-block:: console73 74 $ tools/testing/selftests/bpf/vmtest.sh -h75 76In case of linker errors when running selftests, try using static linking:77 78.. code-block:: console79 80 $ LDLIBS=-static PKG_CONFIG='pkg-config --static' vmtest.sh81 82.. note:: Some distros may not support static linking.83 84.. note:: The script uses pahole and clang based on host environment setting.85 If you want to change pahole and llvm, you can change `PATH` environment86 variable in the beginning of script.87 88Running vmtest on RV6489======================90To speed up testing and avoid various dependency issues, it is recommended to91run vmtest in a Docker container. Before running vmtest, we need to prepare92Docker container and local rootfs image. The overall steps are as follows:93 941. Create Docker container as shown in link [0].95 962. Use mkrootfs_debian.sh script [1] to build local rootfs image:97 98.. code-block:: console99 100 $ sudo ./mkrootfs_debian.sh --arch riscv64 --distro noble101 1023. Start Docker container [0] and run vmtest in the container:103 104.. code-block:: console105 106 $ PLATFORM=riscv64 CROSS_COMPILE=riscv64-linux-gnu- \107 tools/testing/selftests/bpf/vmtest.sh \108 -l <path of local rootfs image> -- \109 ./test_progs -d \110 \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \111 | cut -d'#' -f1 \112 | sed -e 's/^[[:space:]]*//' \113 -e 's/[[:space:]]*$//' \114 | tr -s '\n' ',' \115 )\"116 117Link: https://github.com/pulehui/riscv-bpf-vmtest.git [0]118Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [1]119 120Additional information about selftest failures are121documented here.122 123profiler[23] test failures with clang/llvm <12.0.0124==================================================125 126With clang/llvm <12.0.0, the profiler[23] test may fail.127The symptom looks like128 129.. code-block:: c130 131 // r9 is a pointer to map_value132 // r7 is a scalar133 17: bf 96 00 00 00 00 00 00 r6 = r9134 18: 0f 76 00 00 00 00 00 00 r6 += r7135 math between map_value pointer and register with unbounded min value is not allowed136 137 // the instructions below will not be seen in the verifier log138 19: a5 07 01 00 01 01 00 00 if r7 < 257 goto +1139 20: bf 96 00 00 00 00 00 00 r6 = r9140 // r6 is used here141 142The verifier will reject such code with above error.143At insn 18 the r7 is indeed unbounded. The later insn 19 checks the bounds and144the insn 20 undoes map_value addition. It is currently impossible for the145verifier to understand such speculative pointer arithmetic.146Hence `this patch`__ addresses it on the compiler side. It was committed on llvm 12.147 148__ https://github.com/llvm/llvm-project/commit/ddf1864ace484035e3cde5e83b3a31ac81e059c6149 150The corresponding C code151 152.. code-block:: c153 154 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; i++) {155 filepart_length = bpf_probe_read_str(payload, ...);156 if (filepart_length <= MAX_PATH) {157 barrier_var(filepart_length); // workaround158 payload += filepart_length;159 }160 }161 162bpf_iter test failures with clang/llvm 10.0.0163=============================================164 165With clang/llvm 10.0.0, the following two bpf_iter tests failed:166 * ``bpf_iter/ipv6_route``167 * ``bpf_iter/netlink``168 169The symptom for ``bpf_iter/ipv6_route`` looks like170 171.. code-block:: c172 173 2: (79) r8 = *(u64 *)(r1 +8)174 ...175 14: (bf) r2 = r8176 15: (0f) r2 += r1177 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen);178 16: (7b) *(u64 *)(r8 +64) = r2179 only read is supported180 181The symptom for ``bpf_iter/netlink`` looks like182 183.. code-block:: c184 185 ; struct netlink_sock *nlk = ctx->sk;186 2: (79) r7 = *(u64 *)(r1 +8)187 ...188 15: (bf) r2 = r7189 16: (0f) r2 += r1190 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_protocol);191 17: (7b) *(u64 *)(r7 +0) = r2192 only read is supported193 194This is due to a llvm BPF backend bug. `The fix`__195has been pushed to llvm 10.x release branch and will be196available in 10.0.1. The patch is available in llvm 11.0.0 trunk.197 198__ https://github.com/llvm/llvm-project/commit/3cb7e7bf959dcd3b8080986c62e10a75c7af43f0199 200bpf_verif_scale/loop6.bpf.o test failure with Clang 12201======================================================202 203With Clang 12, the following bpf_verif_scale test failed:204 * ``bpf_verif_scale/loop6.bpf.o``205 206The verifier output looks like207 208.. code-block:: c209 210 R1 type=ctx expected=fp211 The sequence of 8193 jumps is too complex.212 213The reason is compiler generating the following code214 215.. code-block:: c216 217 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {218 14: 16 05 40 00 00 00 00 00 if w5 == 0 goto +64 <LBB0_6>219 15: bc 51 00 00 00 00 00 00 w1 = w5220 16: 04 01 00 00 ff ff ff ff w1 += -1221 17: 67 05 00 00 20 00 00 00 r5 <<= 32222 18: 77 05 00 00 20 00 00 00 r5 >>= 32223 19: a6 01 01 00 05 00 00 00 if w1 < 5 goto +1 <LBB0_4>224 20: b7 05 00 00 06 00 00 00 r5 = 6225 00000000000000a8 <LBB0_4>:226 21: b7 02 00 00 00 00 00 00 r2 = 0227 22: b7 01 00 00 00 00 00 00 r1 = 0228 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) {229 23: 7b 1a e0 ff 00 00 00 00 *(u64 *)(r10 - 32) = r1230 24: 7b 5a c0 ff 00 00 00 00 *(u64 *)(r10 - 64) = r5231 232Note that insn #15 has w1 = w5 and w1 is refined later but233r5(w5) is eventually saved on stack at insn #24 for later use.234This cause later verifier failure. The bug has been `fixed`__ in235Clang 13.236 237__ https://github.com/llvm/llvm-project/commit/1959ead525b8830cc8a345f45e1c3ef9902d3229238 239BPF CO-RE-based tests and Clang version240=======================================241 242A set of selftests use BPF target-specific built-ins, which might require243bleeding-edge Clang versions (Clang 12 nightly at this time).244 245Few sub-tests of core_reloc test suit (part of test_progs test runner) require246the following built-ins, listed with corresponding Clang diffs introducing247them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too248old to support them, they shouldn't cause build failures or runtime test249failures:250 251- __builtin_btf_type_id() [0_, 1_, 2_];252- __builtin_preserve_type_info(), __builtin_preserve_enum_value() [3_, 4_].253 254.. _0: https://github.com/llvm/llvm-project/commit/6b01b465388b204d543da3cf49efd6080db094a9255.. _1: https://github.com/llvm/llvm-project/commit/072cde03aaa13a2c57acf62d79876bf79aa1919f256.. _2: https://github.com/llvm/llvm-project/commit/00602ee7ef0bf6c68d690a2bd729c12b95c95c99257.. _3: https://github.com/llvm/llvm-project/commit/6d218b4adb093ff2e9764febbbc89f429412006c258.. _4: https://github.com/llvm/llvm-project/commit/6d6750696400e7ce988d66a1a00e1d0cb32815f8259 260Floating-point tests and Clang version261======================================262 263Certain selftests, e.g. core_reloc, require support for the floating-point264types, which was introduced in `Clang 13`__. The older Clang versions will265either crash when compiling these tests, or generate an incorrect BTF.266 267__ https://github.com/llvm/llvm-project/commit/a7137b238a07d9399d3ae96c0b461571bd5aa8b2268 269Kernel function call test and Clang version270===========================================271 272Some selftests (e.g. kfunc_call and bpf_tcp_ca) require a LLVM support273to generate extern function in BTF. It was introduced in `Clang 13`__.274 275Without it, the error from compiling bpf selftests looks like:276 277.. code-block:: console278 279 libbpf: failed to find BTF for extern 'tcp_slow_start' [25] section: -2280 281__ https://github.com/llvm/llvm-project/commit/886f9ff53155075bd5f1e994f17b85d1e1b7470c282 283btf_tag test and Clang version284==============================285 286The btf_tag selftest requires LLVM support to recognize the btf_decl_tag and287btf_type_tag attributes. They are introduced in `Clang 14` [0_, 1_].288The subtests ``btf_type_tag_user_{mod1, mod2, vmlinux}`` also requires289pahole version ``1.23``.290 291Without them, the btf_tag selftest will be skipped and you will observe:292 293.. code-block:: console294 295 #<test_num> btf_tag:SKIP296 297.. _0: https://github.com/llvm/llvm-project/commit/a162b67c98066218d0d00aa13b99afb95d9bb5e6298.. _1: https://github.com/llvm/llvm-project/commit/3466e00716e12e32fdb100e3fcfca5c2b3e8d784299 300Clang dependencies for static linking tests301===========================================302 303linked_vars, linked_maps, and linked_funcs tests depend on `Clang fix`__ to304generate valid BTF information for weak variables. Please make sure you use305Clang that contains the fix.306 307__ https://github.com/llvm/llvm-project/commit/968292cb93198442138128d850fd54dc7edc0035308 309Clang relocation changes310========================311 312Clang 13 patch `clang reloc patch`_ made some changes on relocations such313that existing relocation types are broken into more types and314each new type corresponds to only one way to resolve relocation.315See `kernel llvm reloc`_ for more explanation and some examples.316Using clang 13 to compile old libbpf which has static linker support,317there will be a compilation failure::318 319 libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.bpf.o320 321Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``.322To fix this issue, user newer libbpf.323 324.. Links325.. _clang reloc patch: https://github.com/llvm/llvm-project/commit/6a2ea84600ba4bd3b2733bd8f08f5115eb32164b326.. _kernel llvm reloc: /Documentation/bpf/llvm_reloc.rst327 328Clang dependencies for the u32 spill test (xdpwall)329===================================================330The xdpwall selftest requires a change in `Clang 14`__.331 332Without it, the xdpwall selftest will fail and the error message333from running test_progs will look like:334 335.. code-block:: console336 337 test_xdpwall:FAIL:Does LLVM have https://github.com/llvm/llvm-project/commit/ea72b0319d7b0f0c2fcf41d121afa5d031b319d5? unexpected error: -4007338 339__ https://github.com/llvm/llvm-project/commit/ea72b0319d7b0f0c2fcf41d121afa5d031b319d5340