425 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3.. _stateless_decoder:4 5**************************************************6Memory-to-memory Stateless Video Decoder Interface7**************************************************8 9A stateless decoder is a decoder that works without retaining any kind of state10between processed frames. This means that each frame is decoded independently11of any previous and future frames, and that the client is responsible for12maintaining the decoding state and providing it to the decoder with each13decoding request. This is in contrast to the stateful video decoder interface,14where the hardware and driver maintain the decoding state and all the client15has to do is to provide the raw encoded stream and dequeue decoded frames in16display order.17 18This section describes how user-space ("the client") is expected to communicate19with stateless decoders in order to successfully decode an encoded stream.20Compared to stateful codecs, the decoder/client sequence is simpler, but the21cost of this simplicity is extra complexity in the client which is responsible22for maintaining a consistent decoding state.23 24Stateless decoders make use of the :ref:`media-request-api`. A stateless25decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its26``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS`27are invoked.28 29Depending on the encoded formats supported by the decoder, a single decoded30frame may be the result of several decode requests (for instance, H.264 streams31with multiple slices per frame). Decoders that support such formats must also32expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their33``OUTPUT`` queue.34 35Querying capabilities36=====================37 381. To enumerate the set of coded formats supported by the decoder, the client39 calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.40 41 * The driver must always return the full set of supported ``OUTPUT`` formats,42 irrespective of the format currently set on the ``CAPTURE`` queue.43 44 * Simultaneously, the driver must restrain the set of values returned by45 codec-specific capability controls (such as H.264 profiles) to the set46 actually supported by the hardware.47 482. To enumerate the set of supported raw formats, the client calls49 :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue.50 51 * The driver must return only the formats supported for the format currently52 active on the ``OUTPUT`` queue.53 54 * Depending on the currently set ``OUTPUT`` format, the set of supported raw55 formats may depend on the value of some codec-dependent controls.56 The client is responsible for making sure that these controls are set57 before querying the ``CAPTURE`` queue. Failure to do so will result in the58 default values for these controls being used, and a returned set of formats59 that may not be usable for the media the client is trying to decode.60 613. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported62 resolutions for a given format, passing desired pixel format in63 :c:type:`v4l2_frmsizeenum`'s ``pixel_format``.64 654. Supported profiles and levels for the current ``OUTPUT`` format, if66 applicable, may be queried using their respective controls via67 :c:func:`VIDIOC_QUERYCTRL`.68 69Initialization70==============71 721. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`.73 74 * **Required fields:**75 76 ``type``77 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.78 79 ``pixelformat``80 a coded pixel format.81 82 ``width``, ``height``83 coded width and height parsed from the stream.84 85 other fields86 follow standard semantics.87 88 .. note::89 90 Changing the ``OUTPUT`` format may change the currently set ``CAPTURE``91 format. The driver will derive a new ``CAPTURE`` format from the92 ``OUTPUT`` format being set, including resolution, colorimetry93 parameters, etc. If the client needs a specific ``CAPTURE`` format,94 it must adjust it afterwards.95 962. Call :c:func:`VIDIOC_S_EXT_CTRLS` to set all the controls (parsed headers,97 etc.) required by the ``OUTPUT`` format to enumerate the ``CAPTURE`` formats.98 993. Call :c:func:`VIDIOC_G_FMT` for ``CAPTURE`` queue to get the format for the100 destination buffers parsed/decoded from the bytestream.101 102 * **Required fields:**103 104 ``type``105 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.106 107 * **Returned fields:**108 109 ``width``, ``height``110 frame buffer resolution for the decoded frames.111 112 ``pixelformat``113 pixel format for decoded frames.114 115 ``num_planes`` (for _MPLANE ``type`` only)116 number of planes for pixelformat.117 118 ``sizeimage``, ``bytesperline``119 as per standard semantics; matching frame buffer format.120 121 .. note::122 123 The value of ``pixelformat`` may be any pixel format supported for the124 ``OUTPUT`` format, based on the hardware capabilities. It is suggested125 that the driver chooses the preferred/optimal format for the current126 configuration. For example, a YUV format may be preferred over an RGB127 format, if an additional conversion step would be required for RGB.128 1294. *[optional]* Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on130 the ``CAPTURE`` queue. The client may use this ioctl to discover which131 alternative raw formats are supported for the current ``OUTPUT`` format and132 select one of them via :c:func:`VIDIOC_S_FMT`.133 134 .. note::135 136 The driver will return only formats supported for the currently selected137 ``OUTPUT`` format and currently set controls, even if more formats may be138 supported by the decoder in general.139 140 For example, a decoder may support YUV and RGB formats for141 resolutions 1920x1088 and lower, but only YUV for higher resolutions (due142 to hardware limitations). After setting a resolution of 1920x1088 or lower143 as the ``OUTPUT`` format, :c:func:`VIDIOC_ENUM_FMT` may return a set of144 YUV and RGB pixel formats, but after setting a resolution higher than145 1920x1088, the driver will not return RGB pixel formats, since they are146 unsupported for this resolution.147 1485. *[optional]* Choose a different ``CAPTURE`` format than suggested via149 :c:func:`VIDIOC_S_FMT` on ``CAPTURE`` queue. It is possible for the client to150 choose a different format than selected/suggested by the driver in151 :c:func:`VIDIOC_G_FMT`.152 153 * **Required fields:**154 155 ``type``156 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.157 158 ``pixelformat``159 a raw pixel format.160 161 ``width``, ``height``162 frame buffer resolution of the decoded stream; typically unchanged from163 what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different164 if the hardware supports composition and/or scaling.165 166 After performing this step, the client must perform step 3 again in order167 to obtain up-to-date information about the buffers size and layout.168 1696. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on170 ``OUTPUT`` queue.171 172 * **Required fields:**173 174 ``count``175 requested number of buffers to allocate; greater than zero.176 177 ``type``178 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.179 180 ``memory``181 follows standard semantics.182 183 * **Returned fields:**184 185 ``count``186 actual number of buffers allocated.187 188 * If required, the driver will adjust ``count`` to be equal or bigger to the189 minimum of required number of ``OUTPUT`` buffers for the given format and190 requested count. The client must check this value after the ioctl returns191 to get the actual number of buffers allocated.192 1937. Allocate destination (raw format) buffers via :c:func:`VIDIOC_REQBUFS` on the194 ``CAPTURE`` queue.195 196 * **Required fields:**197 198 ``count``199 requested number of buffers to allocate; greater than zero. The client200 is responsible for deducing the minimum number of buffers required201 for the stream to be properly decoded (taking e.g. reference frames202 into account) and pass an equal or bigger number.203 204 ``type``205 a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.206 207 ``memory``208 follows standard semantics. ``V4L2_MEMORY_USERPTR`` is not supported209 for ``CAPTURE`` buffers.210 211 * **Returned fields:**212 213 ``count``214 adjusted to allocated number of buffers, in case the codec requires215 more buffers than requested.216 217 * The driver must adjust count to the minimum of required number of218 ``CAPTURE`` buffers for the current format, stream configuration and219 requested count. The client must check this value after the ioctl220 returns to get the number of buffers allocated.221 2228. Allocate requests (likely one per ``OUTPUT`` buffer) via223 :c:func:`MEDIA_IOC_REQUEST_ALLOC` on the media device.224 2259. Start streaming on both ``OUTPUT`` and ``CAPTURE`` queues via226 :c:func:`VIDIOC_STREAMON`.227 228Decoding229========230 231For each frame, the client is responsible for submitting at least one request to232which the following is attached:233 234* The amount of encoded data expected by the codec for its current235 configuration, as a buffer submitted to the ``OUTPUT`` queue. Typically, this236 corresponds to one frame worth of encoded data, but some formats may allow (or237 require) different amounts per unit.238* All the metadata needed to decode the submitted encoded data, in the form of239 controls relevant to the format being decoded.240 241The amount of data and contents of the source ``OUTPUT`` buffer, as well as the242controls that must be set on the request, depend on the active coded pixel243format and might be affected by codec-specific extended controls, as stated in244documentation of each format.245 246If there is a possibility that the decoded frame will require one or more247decode requests after the current one in order to be produced, then the client248must set the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag on the ``OUTPUT``249buffer. This will result in the (potentially partially) decoded ``CAPTURE``250buffer not being made available for dequeueing, and reused for the next decode251request if the timestamp of the next ``OUTPUT`` buffer has not changed.252 253A typical frame would thus be decoded using the following sequence:254 2551. Queue an ``OUTPUT`` buffer containing one unit of encoded bytestream data for256 the decoding request, using :c:func:`VIDIOC_QBUF`.257 258 * **Required fields:**259 260 ``index``261 index of the buffer being queued.262 263 ``type``264 type of the buffer.265 266 ``bytesused``267 number of bytes taken by the encoded data frame in the buffer.268 269 ``flags``270 the ``V4L2_BUF_FLAG_REQUEST_FD`` flag must be set. Additionally, if271 we are not sure that the current decode request is the last one needed272 to produce a fully decoded frame, then273 ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.274 275 ``request_fd``276 must be set to the file descriptor of the decoding request.277 278 ``timestamp``279 must be set to a unique value per frame. This value will be propagated280 into the decoded frame's buffer and can also be used to use this frame281 as the reference of another. If using multiple decode requests per282 frame, then the timestamps of all the ``OUTPUT`` buffers for a given283 frame must be identical. If the timestamp changes, then the currently284 held ``CAPTURE`` buffer will be made available for dequeuing and the285 current request will work on a new ``CAPTURE`` buffer.286 2872. Set the codec-specific controls for the decoding request, using288 :c:func:`VIDIOC_S_EXT_CTRLS`.289 290 * **Required fields:**291 292 ``which``293 must be ``V4L2_CTRL_WHICH_REQUEST_VAL``.294 295 ``request_fd``296 must be set to the file descriptor of the decoding request.297 298 other fields299 other fields are set as usual when setting controls. The ``controls``300 array must contain all the codec-specific controls required to decode301 a frame.302 303 .. note::304 305 It is possible to specify the controls in different invocations of306 :c:func:`VIDIOC_S_EXT_CTRLS`, or to overwrite a previously set control, as307 long as ``request_fd`` and ``which`` are properly set. The controls state308 at the moment of request submission is the one that will be considered.309 310 .. note::311 312 The order in which steps 1 and 2 take place is interchangeable.313 3143. Submit the request by invoking :c:func:`MEDIA_REQUEST_IOC_QUEUE` on the315 request FD.316 317 If the request is submitted without an ``OUTPUT`` buffer, or if some of the318 required controls are missing from the request, then319 :c:func:`MEDIA_REQUEST_IOC_QUEUE` will return ``-ENOENT``. If more than one320 ``OUTPUT`` buffer is queued, then it will return ``-EINVAL``.321 :c:func:`MEDIA_REQUEST_IOC_QUEUE` returning non-zero means that no322 ``CAPTURE`` buffer will be produced for this request.323 324``CAPTURE`` buffers must not be part of the request, and are queued325independently. They are returned in decode order (i.e. the same order as coded326frames were submitted to the ``OUTPUT`` queue).327 328Runtime decoding errors are signaled by the dequeued ``CAPTURE`` buffers329carrying the ``V4L2_BUF_FLAG_ERROR`` flag. If a decoded reference frame has an330error, then all following decoded frames that refer to it also have the331``V4L2_BUF_FLAG_ERROR`` flag set, although the decoder will still try to332produce (likely corrupted) frames.333 334Buffer management while decoding335================================336Contrary to stateful decoders, a stateless decoder does not perform any kind of337buffer management: it only guarantees that dequeued ``CAPTURE`` buffers can be338used by the client for as long as they are not queued again. "Used" here339encompasses using the buffer for compositing or display.340 341A dequeued capture buffer can also be used as the reference frame of another342buffer.343 344A frame is specified as reference by converting its timestamp into nanoseconds,345and storing it into the relevant member of a codec-dependent control structure.346The :c:func:`v4l2_timeval_to_ns` function must be used to perform that347conversion. The timestamp of a frame can be used to reference it as soon as all348its units of encoded data are successfully submitted to the ``OUTPUT`` queue.349 350A decoded buffer containing a reference frame must not be reused as a decoding351target until all the frames referencing it have been decoded. The safest way to352achieve this is to refrain from queueing a reference buffer until all the353decoded frames referencing it have been dequeued. However, if the driver can354guarantee that buffers queued to the ``CAPTURE`` queue are processed in queued355order, then user-space can take advantage of this guarantee and queue a356reference buffer when the following conditions are met:357 3581. All the requests for frames affected by the reference frame have been359 queued, and360 3612. A sufficient number of ``CAPTURE`` buffers to cover all the decoded362 referencing frames have been queued.363 364When queuing a decoding request, the driver will increase the reference count of365all the resources associated with reference frames. This means that the client366can e.g. close the DMABUF file descriptors of reference frame buffers if it367won't need them afterwards.368 369Seeking370=======371In order to seek, the client just needs to submit requests using input buffers372corresponding to the new stream position. It must however be aware that373resolution may have changed and follow the dynamic resolution change sequence in374that case. Also depending on the codec used, picture parameters (e.g. SPS/PPS375for H.264) may have changed and the client is responsible for making sure that a376valid state is sent to the decoder.377 378The client is then free to ignore any returned ``CAPTURE`` buffer that comes379from the pre-seek position.380 381Pausing382=======383 384In order to pause, the client can just cease queuing buffers onto the ``OUTPUT``385queue. Without source bytestream data, there is no data to process and the codec386will remain idle.387 388Dynamic resolution change389=========================390 391If the client detects a resolution change in the stream, it will need to perform392the initialization sequence again with the new resolution:393 3941. If the last submitted request resulted in a ``CAPTURE`` buffer being395 held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the396 last frame is not available on the ``CAPTURE`` queue. In this case, a397 ``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver398 dequeue the held ``CAPTURE`` buffer.399 4002. Wait until all submitted requests have completed and dequeue the401 corresponding output buffers.402 4033. Call :c:func:`VIDIOC_STREAMOFF` on both the ``OUTPUT`` and ``CAPTURE``404 queues.405 4064. Free all ``CAPTURE`` buffers by calling :c:func:`VIDIOC_REQBUFS` on the407 ``CAPTURE`` queue with a buffer count of zero.408 4095. Perform the initialization sequence again (minus the allocation of410 ``OUTPUT`` buffers), with the new resolution set on the ``OUTPUT`` queue.411 Note that due to resolution constraints, a different format may need to be412 picked on the ``CAPTURE`` queue.413 414Drain415=====416 417If the last submitted request resulted in a ``CAPTURE`` buffer being418held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the419last frame is not available on the ``CAPTURE`` queue. In this case, a420``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver421dequeue the held ``CAPTURE`` buffer.422 423After that, in order to drain the stream on a stateless decoder, the client424just needs to wait until all the submitted requests are completed.425