342 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3Media Controller devices4------------------------5 6Media Controller7~~~~~~~~~~~~~~~~8 9The media controller userspace API is documented in10:ref:`the Media Controller uAPI book <media_controller>`. This document focus11on the kernel-side implementation of the media framework.12 13Abstract media device model14^^^^^^^^^^^^^^^^^^^^^^^^^^^15 16Discovering a device internal topology, and configuring it at runtime, is one17of the goals of the media framework. To achieve this, hardware devices are18modelled as an oriented graph of building blocks called entities connected19through pads.20 21An entity is a basic media hardware building block. It can correspond to22a large variety of logical blocks such as physical hardware devices23(CMOS sensor for instance), logical hardware devices (a building block24in a System-on-Chip image processing pipeline), DMA channels or physical25connectors.26 27A pad is a connection endpoint through which an entity can interact with28other entities. Data (not restricted to video) produced by an entity29flows from the entity's output to one or more entity inputs. Pads should30not be confused with physical pins at chip boundaries.31 32A link is a point-to-point oriented connection between two pads, either33on the same entity or on different entities. Data flows from a source34pad to a sink pad.35 36Media device37^^^^^^^^^^^^38 39A media device is represented by a struct media_device40instance, defined in ``include/media/media-device.h``.41Allocation of the structure is handled by the media device driver, usually by42embedding the :c:type:`media_device` instance in a larger driver-specific43structure.44 45Drivers initialise media device instances by calling46:c:func:`media_device_init()`. After initialising a media device instance, it is47registered by calling :c:func:`__media_device_register()` via the macro48``media_device_register()`` and unregistered by calling49:c:func:`media_device_unregister()`. An initialised media device must be50eventually cleaned up by calling :c:func:`media_device_cleanup()`.51 52Note that it is not allowed to unregister a media device instance that was not53previously registered, or clean up a media device instance that was not54previously initialised.55 56Entities57^^^^^^^^58 59Entities are represented by a struct media_entity60instance, defined in ``include/media/media-entity.h``. The structure is usually61embedded into a higher-level structure, such as62:c:type:`v4l2_subdev` or :c:type:`video_device`63instances, although drivers can allocate entities directly.64 65Drivers initialize entity pads by calling66:c:func:`media_entity_pads_init()`.67 68Drivers register entities with a media device by calling69:c:func:`media_device_register_entity()`70and unregistered by calling71:c:func:`media_device_unregister_entity()`.72 73Interfaces74^^^^^^^^^^75 76Interfaces are represented by a77struct media_interface instance, defined in78``include/media/media-entity.h``. Currently, only one type of interface is79defined: a device node. Such interfaces are represented by a80struct media_intf_devnode.81 82Drivers initialize and create device node interfaces by calling83:c:func:`media_devnode_create()`84and remove them by calling:85:c:func:`media_devnode_remove()`.86 87Pads88^^^^89Pads are represented by a struct media_pad instance,90defined in ``include/media/media-entity.h``. Each entity stores its pads in91a pads array managed by the entity driver. Drivers usually embed the array in92a driver-specific structure.93 94Pads are identified by their entity and their 0-based index in the pads95array.96 97Both information are stored in the struct media_pad,98making the struct media_pad pointer the canonical way99to store and pass link references.100 101Pads have flags that describe the pad capabilities and state.102 103``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.104``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.105 106.. note::107 108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must109 be set for each pad.110 111Links112^^^^^113 114Links are represented by a struct media_link instance,115defined in ``include/media/media-entity.h``. There are two types of links:116 117**1. pad to pad links**:118 119Associate two entities via their PADs. Each entity has a list that points120to all links originating at or targeting any of its pads.121A given link is thus stored twice, once in the source entity and once in122the target entity.123 124Drivers create pad to pad links by calling:125:c:func:`media_create_pad_link()` and remove with126:c:func:`media_entity_remove_links()`.127 128**2. interface to entity links**:129 130Associate one interface to a Link.131 132Drivers create interface to entity links by calling:133:c:func:`media_create_intf_link()` and remove with134:c:func:`media_remove_intf_links()`.135 136.. note::137 138 Links can only be created after having both ends already created.139 140Links have flags that describe the link capabilities and state. The141valid values are described at :c:func:`media_create_pad_link()` and142:c:func:`media_create_intf_link()`.143 144Graph traversal145^^^^^^^^^^^^^^^146 147The media framework provides APIs to traverse media graphs, locating connected148entities and links.149 150To iterate over all entities belonging to a media device, drivers can use151the media_device_for_each_entity macro, defined in152``include/media/media-device.h``.153 154.. code-block:: c155 156 struct media_entity *entity;157 158 media_device_for_each_entity(entity, mdev) {159 // entity will point to each entity in turn160 ...161 }162 163Helper functions can be used to find a link between two given pads, or a pad164connected to another pad through an enabled link165(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`,166:c:func:`media_entity_remote_source_pad_unique()` and167:c:func:`media_pad_remote_pad_unique()`).168 169Use count and power handling170^^^^^^^^^^^^^^^^^^^^^^^^^^^^171 172Due to the wide differences between drivers regarding power management173needs, the media controller does not implement power management. However,174the struct media_entity includes a ``use_count``175field that media drivers176can use to track the number of users of every entity for power management177needs.178 179The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by180media drivers and must not be181touched by entity drivers. Access to the field must be protected by the182:c:type:`media_device`.\ ``graph_mutex`` lock.183 184Links setup185^^^^^^^^^^^186 187Link properties can be modified at runtime by calling188:c:func:`media_entity_setup_link()`.189 190Pipelines and media streams191^^^^^^^^^^^^^^^^^^^^^^^^^^^192 193A media stream is a stream of pixels or metadata originating from one or more194source devices (such as a sensors) and flowing through media entity pads195towards the final sinks. The stream can be modified on the route by the196devices (e.g. scaling or pixel format conversions), or it can be split into197multiple branches, or multiple branches can be merged.198 199A media pipeline is a set of media streams which are interdependent. This200interdependency can be caused by the hardware (e.g. configuration of a second201stream cannot be changed if the first stream has been enabled) or by the driver202due to the software design. Most commonly a media pipeline consists of a single203stream which does not branch.204 205When starting streaming, drivers must notify all entities in the pipeline to206prevent link states from being modified during streaming by calling207:c:func:`media_pipeline_start()`.208 209The function will mark all the pads which are part of the pipeline as streaming.210 211The struct media_pipeline instance pointed to by the pipe argument will be212stored in every pad in the pipeline. Drivers should embed the struct213media_pipeline in higher-level pipeline structures and can then access the214pipeline through the struct media_pad pipe field.215 216Calls to :c:func:`media_pipeline_start()` can be nested.217The pipeline pointer must be identical for all nested calls to the function.218 219:c:func:`media_pipeline_start()` may return an error. In that case,220it will clean up any of the changes it did by itself.221 222When stopping the stream, drivers must notify the entities with223:c:func:`media_pipeline_stop()`.224 225If multiple calls to :c:func:`media_pipeline_start()` have been226made the same number of :c:func:`media_pipeline_stop()` calls227are required to stop streaming.228The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last229nested stop call.230 231Link configuration will fail with ``-EBUSY`` by default if either end of the232link is a streaming entity. Links that can be modified while streaming must233be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.234 235If other operations need to be disallowed on streaming entities (such as236changing entities configuration parameters) drivers can explicitly check the237media_entity stream_count field to find out if an entity is streaming. This238operation must be done with the media_device graph_mutex held.239 240Link validation241^^^^^^^^^^^^^^^242 243Link validation is performed by :c:func:`media_pipeline_start()`244for any entity which has sink pads in the pipeline. The245:c:type:`media_entity`.\ ``link_validate()`` callback is used for that246purpose. In ``link_validate()`` callback, entity driver should check247that the properties of the source pad of the connected entity and its own248sink pad match. It is up to the type of the entity (and in the end, the249properties of the hardware) what matching actually means.250 251Subsystems should facilitate link validation by providing subsystem specific252helper functions to provide easy access for commonly needed information, and253in the end provide a way to use driver-specific callbacks.254 255Pipeline traversal256^^^^^^^^^^^^^^^^^^257 258Once a pipeline has been constructed with :c:func:`media_pipeline_start()`,259drivers can iterate over entities or pads in the pipeline with the260:c:macro:´media_pipeline_for_each_entity` and261:c:macro:´media_pipeline_for_each_pad` macros. Iterating over pads is262straightforward:263 264.. code-block:: c265 266 media_pipeline_pad_iter iter;267 struct media_pad *pad;268 269 media_pipeline_for_each_pad(pipe, &iter, pad) {270 /* 'pad' will point to each pad in turn */271 ...272 }273 274To iterate over entities, the iterator needs to be initialized and cleaned up275as an additional steps:276 277.. code-block:: c278 279 media_pipeline_entity_iter iter;280 struct media_entity *entity;281 int ret;282 283 ret = media_pipeline_entity_iter_init(pipe, &iter);284 if (ret)285 ...;286 287 media_pipeline_for_each_entity(pipe, &iter, entity) {288 /* 'entity' will point to each entity in turn */289 ...290 }291 292 media_pipeline_entity_iter_cleanup(&iter);293 294Media Controller Device Allocator API295^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^296 297When the media device belongs to more than one driver, the shared media298device is allocated with the shared struct device as the key for look ups.299 300The shared media device should stay in registered state until the last301driver unregisters it. In addition, the media device should be released when302all the references are released. Each driver gets a reference to the media303device during probe, when it allocates the media device. If media device is304already allocated, the allocate API bumps up the refcount and returns the305existing media device. The driver puts the reference back in its disconnect306routine when it calls :c:func:`media_device_delete()`.307 308The media device is unregistered and cleaned up from the kref put handler to309ensure that the media device stays in registered state until the last driver310unregisters the media device.311 312**Driver Usage**313 314Drivers should use the appropriate media-core routines to manage the shared315media device life-time handling the two states:3161. allocate -> register -> delete3172. get reference to already registered device -> delete318 319call :c:func:`media_device_delete()` routine to make sure the shared media320device delete is handled correctly.321 322**driver probe:**323Call :c:func:`media_device_usb_allocate()` to allocate or get a reference324Call :c:func:`media_device_register()`, if media devnode isn't registered325 326**driver disconnect:**327Call :c:func:`media_device_delete()` to free the media_device. Freeing is328handled by the kref put handler.329 330API Definitions331^^^^^^^^^^^^^^^332 333.. kernel-doc:: include/media/media-device.h334 335.. kernel-doc:: include/media/media-devnode.h336 337.. kernel-doc:: include/media/media-entity.h338 339.. kernel-doc:: include/media/media-request.h340 341.. kernel-doc:: include/media/media-dev-allocator.h342