Chapter 8: Integrating with V4L2 Async and Media Controller Frameworks

Over time, media support has become a must and a saling argument for System on Chips (SoCs), which keep becoming more and more complex. The complexity of those media IP cores is such that grabbing sensor data requires a whole pipeline (made of several sub-devices) to be set up by the software. The asynchronous nature of a device tree-based system means the setup and probing of those sub-devices are not straightforward. Thus entered the async framework, which addresses the unordered probing of sub-devices in order for the media device to be popped on time, when all of the media sub-devices are ready. Last but not least, because of the complexity of the media pipe, it became necessary to find a way to ease the configuration of the sub-devices it is made of. Thus came the media controller framework, which wraps the whole media pipe in a single element, the media device. It comes with some abstractions, one of which...

Technical requirements

In this chapter, you'll need the following elements:

• Advanced computer architecture knowledge and C programming skills
• Linux kernel v4.19.X sources, available at https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/refs/tags

The V4L2 async interface and the concept of graph binding

So far, with V4L2 driver development, we have not actually dealt with the probing order. That being said, we considered the synchronous approach, where bridge device drivers register devices for all sub-devices synchronously during their probing. However, this approach cannot be used with intrinsically asynchronous and unordered device registration systems, such as the flattened device tree. To address this, what we currently call the async interface has been introduced.

With this new approach, bridge drivers register lists of sub-device descriptors and notifier callbacks, and sub-device drivers register sub-devices that they are about to probe or have successfully probed. The async core will take care of matching sub-devices against hardware descriptors and calling bridge driver callbacks when matches are found. Another callback is called when the sub-device is unregistered. The async subsystem relies on device declaration...

The Linux media controller framework

Media devices turn out to be very complex, involving several IP blocks of the SoC and thus requiring video stream (re)routing.

Now, let's consider a case where we have a much more sophisticated SoC made of two more on-chip sub-devices – let's say a resizer and an image converter, called baz and biz.

In the previous example in the V4L2 async section, the setup was made up of one bridge device and one sub-device (the fact that it is off-chip does not matter), the camera sensor. This was quite straightforward. Luckily, things worked. But what if now we have to route the stream through the image converter or the image resizer, or even through both IPs? Or, say we have to switch from one to the other (dynamically)?

We could achieve this either via sysfs or ioctls, but this would have the following problems:

• It would be too ugly (no doubt) and probably buggy.
• It would be too hard (a lot of work).
• It would be deeply...

Summary

In this chapter, we went through the V4L2 asynchronous interface, which eases video bridge and sub-device driver probing. This is useful for intrinsically asynchronous and unordered device registration systems, such as flattened device tree driver probing. Moreover, we dealt with the media controller framework, which allows leveraging V4L2 video pipelines. What we have seen so far lies in the kernel space.

In the next chapter, we will see how to deal with V4L2 devices from user space, thus leveraging features exposed by their device drivers.