679 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3.. _kfuncs-header-label:4 5=============================6BPF Kernel Functions (kfuncs)7=============================8 91. Introduction10===============11 12BPF Kernel Functions or more commonly known as kfuncs are functions in the Linux13kernel which are exposed for use by BPF programs. Unlike normal BPF helpers,14kfuncs do not have a stable interface and can change from one kernel release to15another. Hence, BPF programs need to be updated in response to changes in the16kernel. See :ref:`BPF_kfunc_lifecycle_expectations` for more information.17 182. Defining a kfunc19===================20 21There are two ways to expose a kernel function to BPF programs, either make an22existing function in the kernel visible, or add a new wrapper for BPF. In both23cases, care must be taken that BPF program can only call such function in a24valid context. To enforce this, visibility of a kfunc can be per program type.25 26If you are not creating a BPF wrapper for existing kernel function, skip ahead27to :ref:`BPF_kfunc_nodef`.28 292.1 Creating a wrapper kfunc30----------------------------31 32When defining a wrapper kfunc, the wrapper function should have extern linkage.33This prevents the compiler from optimizing away dead code, as this wrapper kfunc34is not invoked anywhere in the kernel itself. It is not necessary to provide a35prototype in a header for the wrapper kfunc.36 37An example is given below::38 39 /* Disables missing prototype warnings */40 __bpf_kfunc_start_defs();41 42 __bpf_kfunc struct task_struct *bpf_find_get_task_by_vpid(pid_t nr)43 {44 return find_get_task_by_vpid(nr);45 }46 47 __bpf_kfunc_end_defs();48 49A wrapper kfunc is often needed when we need to annotate parameters of the50kfunc. Otherwise one may directly make the kfunc visible to the BPF program by51registering it with the BPF subsystem. See :ref:`BPF_kfunc_nodef`.52 532.2 Annotating kfunc parameters54-------------------------------55 56Similar to BPF helpers, there is sometime need for additional context required57by the verifier to make the usage of kernel functions safer and more useful.58Hence, we can annotate a parameter by suffixing the name of the argument of the59kfunc with a __tag, where tag may be one of the supported annotations.60 612.2.1 __sz Annotation62---------------------63 64This annotation is used to indicate a memory and size pair in the argument list.65An example is given below::66 67 __bpf_kfunc void bpf_memzero(void *mem, int mem__sz)68 {69 ...70 }71 72Here, the verifier will treat first argument as a PTR_TO_MEM, and second73argument as its size. By default, without __sz annotation, the size of the type74of the pointer is used. Without __sz annotation, a kfunc cannot accept a void75pointer.76 772.2.2 __k Annotation78--------------------79 80This annotation is only understood for scalar arguments, where it indicates that81the verifier must check the scalar argument to be a known constant, which does82not indicate a size parameter, and the value of the constant is relevant to the83safety of the program.84 85An example is given below::86 87 __bpf_kfunc void *bpf_obj_new(u32 local_type_id__k, ...)88 {89 ...90 }91 92Here, bpf_obj_new uses local_type_id argument to find out the size of that type93ID in program's BTF and return a sized pointer to it. Each type ID will have a94distinct size, hence it is crucial to treat each such call as distinct when95values don't match during verifier state pruning checks.96 97Hence, whenever a constant scalar argument is accepted by a kfunc which is not a98size parameter, and the value of the constant matters for program safety, __k99suffix should be used.100 1012.2.3 __uninit Annotation102-------------------------103 104This annotation is used to indicate that the argument will be treated as105uninitialized.106 107An example is given below::108 109 __bpf_kfunc int bpf_dynptr_from_skb(..., struct bpf_dynptr_kern *ptr__uninit)110 {111 ...112 }113 114Here, the dynptr will be treated as an uninitialized dynptr. Without this115annotation, the verifier will reject the program if the dynptr passed in is116not initialized.117 1182.2.4 __opt Annotation119-------------------------120 121This annotation is used to indicate that the buffer associated with an __sz or __szk122argument may be null. If the function is passed a nullptr in place of the buffer,123the verifier will not check that length is appropriate for the buffer. The kfunc is124responsible for checking if this buffer is null before using it.125 126An example is given below::127 128 __bpf_kfunc void *bpf_dynptr_slice(..., void *buffer__opt, u32 buffer__szk)129 {130 ...131 }132 133Here, the buffer may be null. If buffer is not null, it at least of size buffer_szk.134Either way, the returned buffer is either NULL, or of size buffer_szk. Without this135annotation, the verifier will reject the program if a null pointer is passed in with136a nonzero size.137 1382.2.5 __str Annotation139----------------------------140This annotation is used to indicate that the argument is a constant string.141 142An example is given below::143 144 __bpf_kfunc bpf_get_file_xattr(..., const char *name__str, ...)145 {146 ...147 }148 149In this case, ``bpf_get_file_xattr()`` can be called as::150 151 bpf_get_file_xattr(..., "xattr_name", ...);152 153Or::154 155 const char name[] = "xattr_name"; /* This need to be global */156 int BPF_PROG(...)157 {158 ...159 bpf_get_file_xattr(..., name, ...);160 ...161 }162 163.. _BPF_kfunc_nodef:164 1652.3 Using an existing kernel function166-------------------------------------167 168When an existing function in the kernel is fit for consumption by BPF programs,169it can be directly registered with the BPF subsystem. However, care must still170be taken to review the context in which it will be invoked by the BPF program171and whether it is safe to do so.172 1732.4 Annotating kfuncs174---------------------175 176In addition to kfuncs' arguments, verifier may need more information about the177type of kfunc(s) being registered with the BPF subsystem. To do so, we define178flags on a set of kfuncs as follows::179 180 BTF_KFUNCS_START(bpf_task_set)181 BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)182 BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)183 BTF_KFUNCS_END(bpf_task_set)184 185This set encodes the BTF ID of each kfunc listed above, and encodes the flags186along with it. Ofcourse, it is also allowed to specify no flags.187 188kfunc definitions should also always be annotated with the ``__bpf_kfunc``189macro. This prevents issues such as the compiler inlining the kfunc if it's a190static kernel function, or the function being elided in an LTO build as it's191not used in the rest of the kernel. Developers should not manually add192annotations to their kfunc to prevent these issues. If an annotation is193required to prevent such an issue with your kfunc, it is a bug and should be194added to the definition of the macro so that other kfuncs are similarly195protected. An example is given below::196 197 __bpf_kfunc struct task_struct *bpf_get_task_pid(s32 pid)198 {199 ...200 }201 2022.4.1 KF_ACQUIRE flag203---------------------204 205The KF_ACQUIRE flag is used to indicate that the kfunc returns a pointer to a206refcounted object. The verifier will then ensure that the pointer to the object207is eventually released using a release kfunc, or transferred to a map using a208referenced kptr (by invoking bpf_kptr_xchg). If not, the verifier fails the209loading of the BPF program until no lingering references remain in all possible210explored states of the program.211 2122.4.2 KF_RET_NULL flag213----------------------214 215The KF_RET_NULL flag is used to indicate that the pointer returned by the kfunc216may be NULL. Hence, it forces the user to do a NULL check on the pointer217returned from the kfunc before making use of it (dereferencing or passing to218another helper). This flag is often used in pairing with KF_ACQUIRE flag, but219both are orthogonal to each other.220 2212.4.3 KF_RELEASE flag222---------------------223 224The KF_RELEASE flag is used to indicate that the kfunc releases the pointer225passed in to it. There can be only one referenced pointer that can be passed226in. All copies of the pointer being released are invalidated as a result of227invoking kfunc with this flag. KF_RELEASE kfuncs automatically receive the228protection afforded by the KF_TRUSTED_ARGS flag described below.229 2302.4.4 KF_TRUSTED_ARGS flag231--------------------------232 233The KF_TRUSTED_ARGS flag is used for kfuncs taking pointer arguments. It234indicates that the all pointer arguments are valid, and that all pointers to235BTF objects have been passed in their unmodified form (that is, at a zero236offset, and without having been obtained from walking another pointer, with one237exception described below).238 239There are two types of pointers to kernel objects which are considered "valid":240 2411. Pointers which are passed as tracepoint or struct_ops callback arguments.2422. Pointers which were returned from a KF_ACQUIRE kfunc.243 244Pointers to non-BTF objects (e.g. scalar pointers) may also be passed to245KF_TRUSTED_ARGS kfuncs, and may have a non-zero offset.246 247The definition of "valid" pointers is subject to change at any time, and has248absolutely no ABI stability guarantees.249 250As mentioned above, a nested pointer obtained from walking a trusted pointer is251no longer trusted, with one exception. If a struct type has a field that is252guaranteed to be valid (trusted or rcu, as in KF_RCU description below) as long253as its parent pointer is valid, the following macros can be used to express254that to the verifier:255 256* ``BTF_TYPE_SAFE_TRUSTED``257* ``BTF_TYPE_SAFE_RCU``258* ``BTF_TYPE_SAFE_RCU_OR_NULL``259 260For example,261 262.. code-block:: c263 264 BTF_TYPE_SAFE_TRUSTED(struct socket) {265 struct sock *sk;266 };267 268or269 270.. code-block:: c271 272 BTF_TYPE_SAFE_RCU(struct task_struct) {273 const cpumask_t *cpus_ptr;274 struct css_set __rcu *cgroups;275 struct task_struct __rcu *real_parent;276 struct task_struct *group_leader;277 };278 279In other words, you must:280 2811. Wrap the valid pointer type in a ``BTF_TYPE_SAFE_*`` macro.282 2832. Specify the type and name of the valid nested field. This field must match284 the field in the original type definition exactly.285 286A new type declared by a ``BTF_TYPE_SAFE_*`` macro also needs to be emitted so287that it appears in BTF. For example, ``BTF_TYPE_SAFE_TRUSTED(struct socket)``288is emitted in the ``type_is_trusted()`` function as follows:289 290.. code-block:: c291 292 BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED(struct socket));293 294 2952.4.5 KF_SLEEPABLE flag296-----------------------297 298The KF_SLEEPABLE flag is used for kfuncs that may sleep. Such kfuncs can only299be called by sleepable BPF programs (BPF_F_SLEEPABLE).300 3012.4.6 KF_DESTRUCTIVE flag302--------------------------303 304The KF_DESTRUCTIVE flag is used to indicate functions calling which is305destructive to the system. For example such a call can result in system306rebooting or panicking. Due to this additional restrictions apply to these307calls. At the moment they only require CAP_SYS_BOOT capability, but more can be308added later.309 3102.4.7 KF_RCU flag311-----------------312 313The KF_RCU flag is a weaker version of KF_TRUSTED_ARGS. The kfuncs marked with314KF_RCU expect either PTR_TRUSTED or MEM_RCU arguments. The verifier guarantees315that the objects are valid and there is no use-after-free. The pointers are not316NULL, but the object's refcount could have reached zero. The kfuncs need to317consider doing refcnt != 0 check, especially when returning a KF_ACQUIRE318pointer. Note as well that a KF_ACQUIRE kfunc that is KF_RCU should very likely319also be KF_RET_NULL.320 321.. _KF_deprecated_flag:322 3232.4.8 KF_DEPRECATED flag324------------------------325 326The KF_DEPRECATED flag is used for kfuncs which are scheduled to be327changed or removed in a subsequent kernel release. A kfunc that is328marked with KF_DEPRECATED should also have any relevant information329captured in its kernel doc. Such information typically includes the330kfunc's expected remaining lifespan, a recommendation for new331functionality that can replace it if any is available, and possibly a332rationale for why it is being removed.333 334Note that while on some occasions, a KF_DEPRECATED kfunc may continue to be335supported and have its KF_DEPRECATED flag removed, it is likely to be far more336difficult to remove a KF_DEPRECATED flag after it's been added than it is to337prevent it from being added in the first place. As described in338:ref:`BPF_kfunc_lifecycle_expectations`, users that rely on specific kfuncs are339encouraged to make their use-cases known as early as possible, and participate340in upstream discussions regarding whether to keep, change, deprecate, or remove341those kfuncs if and when such discussions occur.342 3432.5 Registering the kfuncs344--------------------------345 346Once the kfunc is prepared for use, the final step to making it visible is347registering it with the BPF subsystem. Registration is done per BPF program348type. An example is shown below::349 350 BTF_KFUNCS_START(bpf_task_set)351 BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)352 BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)353 BTF_KFUNCS_END(bpf_task_set)354 355 static const struct btf_kfunc_id_set bpf_task_kfunc_set = {356 .owner = THIS_MODULE,357 .set = &bpf_task_set,358 };359 360 static int init_subsystem(void)361 {362 return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_task_kfunc_set);363 }364 late_initcall(init_subsystem);365 3662.6 Specifying no-cast aliases with ___init367--------------------------------------------368 369The verifier will always enforce that the BTF type of a pointer passed to a370kfunc by a BPF program, matches the type of pointer specified in the kfunc371definition. The verifier, does, however, allow types that are equivalent372according to the C standard to be passed to the same kfunc arg, even if their373BTF_IDs differ.374 375For example, for the following type definition:376 377.. code-block:: c378 379 struct bpf_cpumask {380 cpumask_t cpumask;381 refcount_t usage;382 };383 384The verifier would allow a ``struct bpf_cpumask *`` to be passed to a kfunc385taking a ``cpumask_t *`` (which is a typedef of ``struct cpumask *``). For386instance, both ``struct cpumask *`` and ``struct bpf_cpmuask *`` can be passed387to bpf_cpumask_test_cpu().388 389In some cases, this type-aliasing behavior is not desired. ``struct390nf_conn___init`` is one such example:391 392.. code-block:: c393 394 struct nf_conn___init {395 struct nf_conn ct;396 };397 398The C standard would consider these types to be equivalent, but it would not399always be safe to pass either type to a trusted kfunc. ``struct400nf_conn___init`` represents an allocated ``struct nf_conn`` object that has401*not yet been initialized*, so it would therefore be unsafe to pass a ``struct402nf_conn___init *`` to a kfunc that's expecting a fully initialized ``struct403nf_conn *`` (e.g. ``bpf_ct_change_timeout()``).404 405In order to accommodate such requirements, the verifier will enforce strict406PTR_TO_BTF_ID type matching if two types have the exact same name, with one407being suffixed with ``___init``.408 409.. _BPF_kfunc_lifecycle_expectations:410 4113. kfunc lifecycle expectations412===============================413 414kfuncs provide a kernel <-> kernel API, and thus are not bound by any of the415strict stability restrictions associated with kernel <-> user UAPIs. This means416they can be thought of as similar to EXPORT_SYMBOL_GPL, and can therefore be417modified or removed by a maintainer of the subsystem they're defined in when418it's deemed necessary.419 420Like any other change to the kernel, maintainers will not change or remove a421kfunc without having a reasonable justification. Whether or not they'll choose422to change a kfunc will ultimately depend on a variety of factors, such as how423widely used the kfunc is, how long the kfunc has been in the kernel, whether an424alternative kfunc exists, what the norm is in terms of stability for the425subsystem in question, and of course what the technical cost is of continuing426to support the kfunc.427 428There are several implications of this:429 430a) kfuncs that are widely used or have been in the kernel for a long time will431 be more difficult to justify being changed or removed by a maintainer. In432 other words, kfuncs that are known to have a lot of users and provide433 significant value provide stronger incentives for maintainers to invest the434 time and complexity in supporting them. It is therefore important for435 developers that are using kfuncs in their BPF programs to communicate and436 explain how and why those kfuncs are being used, and to participate in437 discussions regarding those kfuncs when they occur upstream.438 439b) Unlike regular kernel symbols marked with EXPORT_SYMBOL_GPL, BPF programs440 that call kfuncs are generally not part of the kernel tree. This means that441 refactoring cannot typically change callers in-place when a kfunc changes,442 as is done for e.g. an upstreamed driver being updated in place when a443 kernel symbol is changed.444 445 Unlike with regular kernel symbols, this is expected behavior for BPF446 symbols, and out-of-tree BPF programs that use kfuncs should be considered447 relevant to discussions and decisions around modifying and removing those448 kfuncs. The BPF community will take an active role in participating in449 upstream discussions when necessary to ensure that the perspectives of such450 users are taken into account.451 452c) A kfunc will never have any hard stability guarantees. BPF APIs cannot and453 will not ever hard-block a change in the kernel purely for stability454 reasons. That being said, kfuncs are features that are meant to solve455 problems and provide value to users. The decision of whether to change or456 remove a kfunc is a multivariate technical decision that is made on a457 case-by-case basis, and which is informed by data points such as those458 mentioned above. It is expected that a kfunc being removed or changed with459 no warning will not be a common occurrence or take place without sound460 justification, but it is a possibility that must be accepted if one is to461 use kfuncs.462 4633.1 kfunc deprecation464---------------------465 466As described above, while sometimes a maintainer may find that a kfunc must be467changed or removed immediately to accommodate some changes in their subsystem,468usually kfuncs will be able to accommodate a longer and more measured469deprecation process. For example, if a new kfunc comes along which provides470superior functionality to an existing kfunc, the existing kfunc may be471deprecated for some period of time to allow users to migrate their BPF programs472to use the new one. Or, if a kfunc has no known users, a decision may be made473to remove the kfunc (without providing an alternative API) after some474deprecation period so as to provide users with a window to notify the kfunc475maintainer if it turns out that the kfunc is actually being used.476 477It's expected that the common case will be that kfuncs will go through a478deprecation period rather than being changed or removed without warning. As479described in :ref:`KF_deprecated_flag`, the kfunc framework provides the480KF_DEPRECATED flag to kfunc developers to signal to users that a kfunc has been481deprecated. Once a kfunc has been marked with KF_DEPRECATED, the following482procedure is followed for removal:483 4841. Any relevant information for deprecated kfuncs is documented in the kfunc's485 kernel docs. This documentation will typically include the kfunc's expected486 remaining lifespan, a recommendation for new functionality that can replace487 the usage of the deprecated function (or an explanation as to why no such488 replacement exists), etc.489 4902. The deprecated kfunc is kept in the kernel for some period of time after it491 was first marked as deprecated. This time period will be chosen on a492 case-by-case basis, and will typically depend on how widespread the use of493 the kfunc is, how long it has been in the kernel, and how hard it is to move494 to alternatives. This deprecation time period is "best effort", and as495 described :ref:`above<BPF_kfunc_lifecycle_expectations>`, circumstances may496 sometimes dictate that the kfunc be removed before the full intended497 deprecation period has elapsed.498 4993. After the deprecation period the kfunc will be removed. At this point, BPF500 programs calling the kfunc will be rejected by the verifier.501 5024. Core kfuncs503==============504 505The BPF subsystem provides a number of "core" kfuncs that are potentially506applicable to a wide variety of different possible use cases and programs.507Those kfuncs are documented here.508 5094.1 struct task_struct * kfuncs510-------------------------------511 512There are a number of kfuncs that allow ``struct task_struct *`` objects to be513used as kptrs:514 515.. kernel-doc:: kernel/bpf/helpers.c516 :identifiers: bpf_task_acquire bpf_task_release517 518These kfuncs are useful when you want to acquire or release a reference to a519``struct task_struct *`` that was passed as e.g. a tracepoint arg, or a520struct_ops callback arg. For example:521 522.. code-block:: c523 524 /**525 * A trivial example tracepoint program that shows how to526 * acquire and release a struct task_struct * pointer.527 */528 SEC("tp_btf/task_newtask")529 int BPF_PROG(task_acquire_release_example, struct task_struct *task, u64 clone_flags)530 {531 struct task_struct *acquired;532 533 acquired = bpf_task_acquire(task);534 if (acquired)535 /*536 * In a typical program you'd do something like store537 * the task in a map, and the map will automatically538 * release it later. Here, we release it manually.539 */540 bpf_task_release(acquired);541 return 0;542 }543 544 545References acquired on ``struct task_struct *`` objects are RCU protected.546Therefore, when in an RCU read region, you can obtain a pointer to a task547embedded in a map value without having to acquire a reference:548 549.. code-block:: c550 551 #define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))552 private(TASK) static struct task_struct *global;553 554 /**555 * A trivial example showing how to access a task stored556 * in a map using RCU.557 */558 SEC("tp_btf/task_newtask")559 int BPF_PROG(task_rcu_read_example, struct task_struct *task, u64 clone_flags)560 {561 struct task_struct *local_copy;562 563 bpf_rcu_read_lock();564 local_copy = global;565 if (local_copy)566 /*567 * We could also pass local_copy to kfuncs or helper functions here,568 * as we're guaranteed that local_copy will be valid until we exit569 * the RCU read region below.570 */571 bpf_printk("Global task %s is valid", local_copy->comm);572 else573 bpf_printk("No global task found");574 bpf_rcu_read_unlock();575 576 /* At this point we can no longer reference local_copy. */577 578 return 0;579 }580 581----582 583A BPF program can also look up a task from a pid. This can be useful if the584caller doesn't have a trusted pointer to a ``struct task_struct *`` object that585it can acquire a reference on with bpf_task_acquire().586 587.. kernel-doc:: kernel/bpf/helpers.c588 :identifiers: bpf_task_from_pid589 590Here is an example of it being used:591 592.. code-block:: c593 594 SEC("tp_btf/task_newtask")595 int BPF_PROG(task_get_pid_example, struct task_struct *task, u64 clone_flags)596 {597 struct task_struct *lookup;598 599 lookup = bpf_task_from_pid(task->pid);600 if (!lookup)601 /* A task should always be found, as %task is a tracepoint arg. */602 return -ENOENT;603 604 if (lookup->pid != task->pid) {605 /* bpf_task_from_pid() looks up the task via its606 * globally-unique pid from the init_pid_ns. Thus,607 * the pid of the lookup task should always be the608 * same as the input task.609 */610 bpf_task_release(lookup);611 return -EINVAL;612 }613 614 /* bpf_task_from_pid() returns an acquired reference,615 * so it must be dropped before returning from the616 * tracepoint handler.617 */618 bpf_task_release(lookup);619 return 0;620 }621 6224.2 struct cgroup * kfuncs623--------------------------624 625``struct cgroup *`` objects also have acquire and release functions:626 627.. kernel-doc:: kernel/bpf/helpers.c628 :identifiers: bpf_cgroup_acquire bpf_cgroup_release629 630These kfuncs are used in exactly the same manner as bpf_task_acquire() and631bpf_task_release() respectively, so we won't provide examples for them.632 633----634 635Other kfuncs available for interacting with ``struct cgroup *`` objects are636bpf_cgroup_ancestor() and bpf_cgroup_from_id(), allowing callers to access637the ancestor of a cgroup and find a cgroup by its ID, respectively. Both638return a cgroup kptr.639 640.. kernel-doc:: kernel/bpf/helpers.c641 :identifiers: bpf_cgroup_ancestor642 643.. kernel-doc:: kernel/bpf/helpers.c644 :identifiers: bpf_cgroup_from_id645 646Eventually, BPF should be updated to allow this to happen with a normal memory647load in the program itself. This is currently not possible without more work in648the verifier. bpf_cgroup_ancestor() can be used as follows:649 650.. code-block:: c651 652 /**653 * Simple tracepoint example that illustrates how a cgroup's654 * ancestor can be accessed using bpf_cgroup_ancestor().655 */656 SEC("tp_btf/cgroup_mkdir")657 int BPF_PROG(cgrp_ancestor_example, struct cgroup *cgrp, const char *path)658 {659 struct cgroup *parent;660 661 /* The parent cgroup resides at the level before the current cgroup's level. */662 parent = bpf_cgroup_ancestor(cgrp, cgrp->level - 1);663 if (!parent)664 return -ENOENT;665 666 bpf_printk("Parent id is %d", parent->self.id);667 668 /* Return the parent cgroup that was acquired above. */669 bpf_cgroup_release(parent);670 return 0;671 }672 6734.3 struct cpumask * kfuncs674---------------------------675 676BPF provides a set of kfuncs that can be used to query, allocate, mutate, and677destroy struct cpumask * objects. Please refer to :ref:`cpumasks-header-label`678for more details.679