117 lines · plain
1===========================2InfiniBand Midlayer Locking3===========================4 5 This guide is an attempt to make explicit the locking assumptions6 made by the InfiniBand midlayer. It describes the requirements on7 both low-level drivers that sit below the midlayer and upper level8 protocols that use the midlayer.9 10Sleeping and interrupt context11==============================12 13 With the following exceptions, a low-level driver implementation of14 all of the methods in struct ib_device may sleep. The exceptions15 are any methods from the list:16 17 - create_ah18 - modify_ah19 - query_ah20 - destroy_ah21 - post_send22 - post_recv23 - poll_cq24 - req_notify_cq25 26 which may not sleep and must be callable from any context.27 28 The corresponding functions exported to upper level protocol29 consumers:30 31 - rdma_create_ah32 - rdma_modify_ah33 - rdma_query_ah34 - rdma_destroy_ah35 - ib_post_send36 - ib_post_recv37 - ib_req_notify_cq38 39 are therefore safe to call from any context.40 41 In addition, the function42 43 - ib_dispatch_event44 45 used by low-level drivers to dispatch asynchronous events through46 the midlayer is also safe to call from any context.47 48Reentrancy49----------50 51 All of the methods in struct ib_device exported by a low-level52 driver must be fully reentrant. The low-level driver is required to53 perform all synchronization necessary to maintain consistency, even54 if multiple function calls using the same object are run55 simultaneously.56 57 The IB midlayer does not perform any serialization of function calls.58 59 Because low-level drivers are reentrant, upper level protocol60 consumers are not required to perform any serialization. However,61 some serialization may be required to get sensible results. For62 example, a consumer may safely call ib_poll_cq() on multiple CPUs63 simultaneously. However, the ordering of the work completion64 information between different calls of ib_poll_cq() is not defined.65 66Callbacks67---------68 69 A low-level driver must not perform a callback directly from the70 same callchain as an ib_device method call. For example, it is not71 allowed for a low-level driver to call a consumer's completion event72 handler directly from its post_send method. Instead, the low-level73 driver should defer this callback by, for example, scheduling a74 tasklet to perform the callback.75 76 The low-level driver is responsible for ensuring that multiple77 completion event handlers for the same CQ are not called78 simultaneously. The driver must guarantee that only one CQ event79 handler for a given CQ is running at a time. In other words, the80 following situation is not allowed::81 82 CPU1 CPU283 84 low-level driver ->85 consumer CQ event callback:86 /* ... */87 ib_req_notify_cq(cq, ...);88 low-level driver ->89 /* ... */ consumer CQ event callback:90 /* ... */91 return from CQ event handler92 93 The context in which completion event and asynchronous event94 callbacks run is not defined. Depending on the low-level driver, it95 may be process context, softirq context, or interrupt context.96 Upper level protocol consumers may not sleep in a callback.97 98Hot-plug99--------100 101 A low-level driver announces that a device is ready for use by102 consumers when it calls ib_register_device(), all initialization103 must be complete before this call. The device must remain usable104 until the driver's call to ib_unregister_device() has returned.105 106 A low-level driver must call ib_register_device() and107 ib_unregister_device() from process context. It must not hold any108 semaphores that could cause deadlock if a consumer calls back into109 the driver across these calls.110 111 An upper level protocol consumer may begin using an IB device as112 soon as the add method of its struct ib_client is called for that113 device. A consumer must finish all cleanup and free all resources114 relating to a device before returning from the remove method.115 116 A consumer is permitted to sleep in its add and remove methods.117