Introduction: Not That Type of Advertisement
Hello again! Last time, we talked about implementing multi-plane format support, and now after having everything fully wired up, we confirmed that it compiles and hence it’s time to ship it and move on to the next stage: YCbCr advertisement. Hm? I forgot something? It probably wasn’t important then. Probably. Maybe. :^)
Put simply, advertisement is basically the driver telling applications that a certain feature or extension is supported. This is important as first of all, features obviously can’t be used unless supported, and secondly, Vulkan has plenty of optional features, so advertisement serves as a way for applications to know what is and isn’t supported right away. In our case, what we are concerned with advertising would be all the YCbCr related extensions (text taken from the Vulkan specification):
VK_KHR_sampler_ycbcr_conversion: The main YCbCr extension for Vulkan, promoted to Vulkan 1.1 core. This extension provides the ability to perform specified color space conversions during texture sampling operations for the YCbCr color space natively. It also adds a selection of multi-planar formats, image aspect plane, and the ability to bind memory to the planes of an image collectively or separately.VK_EXT_ycbcr_2plane_444_formats: A secondary YCbCr extension, promoted to Vulkan 1.3 core. This extension adds some YCbCr formats that are in common use for video encode and decode, but were not part of the VK_KHR_sampler_ycbcr_conversion extension.VK_EXT_ycbcr_image_arrays: This extension allows images of a format that requires YCbCr conversion to be created with multiple array layers, which is otherwise restricted.
Advertisement is really simple and straightforward: there are basically a few switches to flip and provided the functionality being advertised is implemented correctly, that’s it. However, along the way, given this project adds new image types, we’ll also have to make some changes to how image features are advertised in order to properly handle multi-plane images. That’s basically it, except we also run into… well, let’s not ruin the surprise, shall we? So now, let’s begin!
.KHR_sampler_ycbcr_conversion = true: Come Get Your VK_KHR_sampler_ycbcr_conversion From The Hot New Driver In Town
As we have established above, advertisement is the driver telling applications that particular features are supported, but how does this work? Drivers have a mechanism for reporting this through functions which fill in Vulkan tables that indicate which features are supported and which are not. To be precise, there are two such tables; one for the extensions the driver supports, and another for the features. This can feel like a weird split, but a driver can support an extension but not all of its features (whether temporarily during development, or because some features are optional), so it’s important to maintain such a split. By default, the tables are filled in with “unsupported” for everything. So essentially, what we need to do here is add in the features we want and set them to true, starting with the extensions table:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
static void
nvk_get_device_extensions(const struct nv_device_info *info,
struct vk_device_extension_table *ext)
{
*ext = (struct vk_device_extension_table) {
/* Many extensions */
.KHR_sampler_ycbcr_conversion = true,
/* Many extensions */
.EXT_ycbcr_2plane_444_formats = true,
.EXT_ycbcr_image_arrays = true,
};
}
The extension table is rather large, so I have omitted the lines that were already there and only added in the relevant lines. Now, we advertise the extensions, so the next step is advertising the features:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
static void
nvk_get_device_features(const struct nv_device_info *info,
struct vk_features *features)
{
*features = (struct vk_features) {
/* A lot of features */
/* Vulkan 1.1 */
.samplerYcbcrConversion = true,
/* Even more features */
/* VK_EXT_ycbcr_2plane_444_formats */
.ycbcr2plane444Formats = true,
/* VK_EXT_ycbcr_image_arrays */
.ycbcrImageArrays = true,
/* One more feature */
};
}
Similar to the extension table, the features table is quite large, so I have focused on the parts we’re concerned with. The extensions we’re enabling here have only one feature to advertise, so there’s nothing to do except enable them and move on.
Format Properties and Feature Flags: The Devil is in the Details
Format properties and feature flags are essentially pieces of information exposed in Vulkan that describe the image and what you can and cannot do with it. This ranges from things like image maximum dimensions and mip levels to information like whether the image can be sampled or not, and the type of filtering you can do on the image. The driver has to specify these correctly otherwise applications may ask for something the hardware doesn’t actually support, or something that simply doesn’t make sense like YCbCr images that cannot be sampled or single-plane disjoint images.
In other words, the driver has two jobs here: fill in the flags and properties correctly, and ensure that incompatible or wrong combinations of flags are reported as unsupported. This was all already handled by the current code, we just needed to adapt that to multi-plane formats, as well as add a few extra checks and flags for YCbCr. Now, YCbCr formats aren’t exclusively multi-plane, but so far we have only implemented support for multi-planar formats, so this is just a starting point that we will tweak later on.
Starting out with format properties in nvk_GetPhysicalDeviceImageFormatProperties2(), we just needed to do a simple check to ensure that the YCbCr image is 2D, as YCbCr formats are 2D only.
1
2
3
4
5
const struct vk_format_ycbcr_info *ycbcr_info =
vk_format_get_ycbcr_info(pImageFormatInfo->format);
if (ycbcr_info && pImageFormatInfo->type != VK_IMAGE_TYPE_2D)
return VK_ERROR_FORMAT_NOT_SUPPORTED;
We will get more into this later, but ycbcr_info is a common (i.e., across all of Mesa) structure that encapsulates information about all YCbCr formats. If the format in question is a YCbCr one, it will contain its number of planes and the individual planes and their formats. If not, it will return NULL.
Afterwards, we set the maximum mip levels and sample counts for YCbCr formats:
1
2
3
4
5
6
7
8
9
10
if(ycbcr_info) {
maxMipLevels = 1;
sampleCounts = VK_SAMPLE_COUNT_1_BIT;
} else {
maxMipLevels = 15;
sampleCounts = VK_SAMPLE_COUNT_1_BIT |
VK_SAMPLE_COUNT_2_BIT |
VK_SAMPLE_COUNT_4_BIT |
VK_SAMPLE_COUNT_8_BIT;
}
Following that, everything gets handled smoothly by the rest of the code; the driver continues filling in other properties (that don’t need special treatment for YCbCr), fills in feature flags, and finally passes in everything at the end.
And speaking of feature flags… Well, multi-planar formats are just a bunch of formats grouped together, right? In other words, we can say that the overall feature flags for a multi-planar format are the intersection of the feature flags of each individual plane. So, we would just extract the number of planes, and loop over each of them and we’re done. And given the current code already does this in a function for a single plane, you can see where this is going…
It’s so obvious!
In other words, if the function detects that the format it’s filling in feature flags for is a YCbCr one, just recursively loop over all the planes, with each iteration taking the intersection of the flags of previous iterations:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
VkFormatFeatureFlags2
nvk_get_image_format_features(struct nvk_physical_device *pdev,
VkFormat vk_format, VkImageTiling tiling)
{
/* Do non-YCbCr stuff */
const struct vk_format_ycbcr_info *ycbcr_info =
vk_format_get_ycbcr_info(vk_format);
if (ycbcr_info && ycbcr_info->n_planes > 1) {
features = ~0ull;
bool cosited_chroma = false;
for (uint8_t plane = 0; plane < ycbcr_info->n_planes; plane++) {
const struct vk_format_ycbcr_plane *plane_info = &ycbcr_info->planes[plane];
features &= nvk_get_image_format_features(pdev, plane_info->format, tiling);
if (plane_info->denominator_scales[0] > 1 ||
plane_info->denominator_scales[1] > 1)
cosited_chroma = true;
}
features |= VK_FORMAT_FEATURE_2_SAMPLED_IMAGE_YCBCR_CONVERSION_LINEAR_FILTER_BIT |
VK_FORMAT_FEATURE_2_MIDPOINT_CHROMA_SAMPLES_BIT |
VK_FORMAT_FEATURE_2_SAMPLED_IMAGE_YCBCR_CONVERSION_SEPARATE_RECONSTRUCTION_FILTER_BIT;
features |= VK_FORMAT_FEATURE_DISJOINT_BIT;
if (cosited_chroma)
features |= VK_FORMAT_FEATURE_COSITED_CHROMA_SAMPLES_BIT;
}
/* Continue doing non-YCbCr stuff */
}
However, while this does work, it’s not exactly ideal. First of all, recursion is always iffy in many regards, doubly so in this case given the base case is a bit questionable. Not just that, but the above style breaks down in face of single-plane YCbCr formats, and while it could have been tweaked to work with it, it’s just better at this point to go for a more conventional approach. In the end, we ended up with two helper functions: one for filling in the feature flags of a plane, and one for the feature flags of a whole image. Single-planar formats would go through the single-plane function, while multi-planar would go through the other one, which would in turn call in the single-plane helper for each of the individual planes. The final code is a bit too long to write here, so if one is interested, it can be found here.
You may notice that there are a few feature flags enabled here as well. Below is a short summary on each, minus two bits we’ll take about in the sampling blog post:
VK_FORMAT_FEATURE_DISJOINT_BIT: Specifies that a multi-planar image can have the VK_IMAGE_CREATE_DISJOINT_BIT set during image creation. In other words, this allows you to create disjoint images using this format.VK_FORMAT_FEATURE_COSITED_CHROMA_SAMPLES_BIT: Allows a downsampled format to have xChromaOffset and/or yChromaOffset of VK_CHROMA_LOCATION_COSITED_EVEN, which means that downsampled chroma samples are aligned with luma samples with even coordinates.VK_FORMAT_FEATURE_2_MIDPOINT_CHROMA_SAMPLES_BIT: Allows a downsampled format to have xChromaOffset and/or yChromaOffset of VK_CHROMA_LOCATION_MIDPOINT, which means that downsampled chroma samples are located half way between each even luma sample and the nearest higher odd luma sample.
Taking care of this ensures that we properly fill in the image format properties that we support. Naturally, then the next step is blocking off the things we don’t support. This part is a bit weird because it’s actually really simple and easy – it’s just adding in a few if conditions to account for corner cases, but at the same time it’s not hard at all to miss something because there are many corner cases and things to keep in mind, and while they’re technically written out in the Vulkan specification, they’re quite buried deep, and hence the title of this section. In fact, I actually ended up missing a fair bit in the end, despite trying to account for everything.
1
2
3
4
5
6
features &= ~(VK_FORMAT_FEATURE_2_BLIT_SRC_BIT |
VK_FORMAT_FEATURE_2_BLIT_DST_BIT |
VK_FORMAT_FEATURE_2_COLOR_ATTACHMENT_BIT |
VK_FORMAT_FEATURE_2_COLOR_ATTACHMENT_BLEND_BIT |
VK_FORMAT_FEATURE_2_STORAGE_IMAGE_BIT);
The first four are rather simple, you can’t use YCbCr formats as color attachments, nor for blitting. As for the last one, Vulkan has two image types: Storage Images, and Sampled Images. Storage Images are images you can do load, store, as well as atomic operations one, while Sampled Images are images you can do sampling operations on. Given that YCbCr formats are for sampling, we need the latter image type rather than the former. Finally:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/* From the Vulkan 1.3.259 spec, VkImageCreateInfo:
*
* VUID-VkImageCreateInfo-imageCreateFormatFeatures-02260
*
* "If format is a multi-planar format, and if imageCreateFormatFeatures
* (as defined in Image Creation Limits) does not contain
* VK_FORMAT_FEATURE_DISJOINT_BIT, then flags must not contain
* VK_IMAGE_CREATE_DISJOINT_BIT"
*
* This is satisfied trivially because we support DISJOINT on all
* multi-plane formats. Also,
*
* VUID-VkImageCreateInfo-format-01577
*
* "If format is not a multi-planar format, and flags does not include
* VK_IMAGE_CREATE_ALIAS_BIT, flags must not contain
* VK_IMAGE_CREATE_DISJOINT_BIT"
*/
if (plane_count == 1 &&
!(pImageFormatInfo->flags & VK_IMAGE_CREATE_ALIAS_BIT) &&
(pImageFormatInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT))
return VK_ERROR_FORMAT_NOT_SUPPORTED;
In other words, if a multi-planar format isn’t defined as supporting DISJOINT, an application cannot create an image using that format with the DISJOINT bit set. However, this doesn’t concern us because we support DISJOINT for all multi-plane formats. As for the second one, an application cannot create a single-plane DISJOINT image without also setting the VK_IMAGE_CREATE_ALIAS_BIT bit. This may seem a bit counter-intuitive (how can a single plane image be disjoint?) but the Vulkan specification for CREATE_ALIAS_BIT expands on this:
VK_IMAGE_CREATE_ALIAS_BIT: […] This flag further specifies that each plane of a disjoint image can share an in-memory non-linear representation with single-plane images, and that a single-plane image can share an in-memory non-linear representation with a plane of a multi-planar disjoint image […].
The T Word
With all that done and out of the way, a simple question remains: advertisement implies that you can do what you are advertising, so why are we advertising the extension relatively early on? The answer is something I neglected to mention except as a note at the end of the previous post: testing. I know, I know, testing software? In 2023? Who does that any more, right?
A tale as old as time
Depending on the task at hand, testing and debugging can be done through a variety of ways; writing dedicated tests/bug replicators, running applications that use or showcase the functionality you are implementing, crying at night in front of your monitor as you find that no matter what you do you can’t replicate that issue, … you get the idea. One important application in particular is the Vulkan Conformance Test Suite (CTS), which includes a large amount of tests for most if not all Vulkan features, with it serving as the benchmark for whether a driver is a conformant implementation and supports these features or not. However, given the way the tests are written, it’s not useful for just conformance, but is also useful for testing out whether your implementation is working properly or not. And indeed, traditionally for new extension implementations, what happens is they’re mainly tested with either the CTS tests for that particular extension alone, or that along with a few hand written tests, depending on how things go.
In our case, the CTS proved to be enough. At this stage, we have implemented only multi-plane format support, so the main tests we can run are the dEQP-Vk.ycbcr.copy.* tests, which test copying from one image to another, with different combinations of formats and image types. Things were fairly straightforward: load the driver, run the CTS with the subset of tests we want, and observe the output. In case of failing tests, crashes, or segfaults (of course), you usually get an idea of where to start; the CTS outputs a log file with all the test cases, and for failures it gives you more information on the failure in question, which usually guides you towards a starting point. Emphasis on usually: for some tests, the error log doesn’t do much beyond telling you that you have a failure, and it can be really annoying at times to tracing the root of the issue, which we’ll see more of in a future post. Additionally, the CTS itself isn’t immune to bugs or oversights, and sometimes things pass when they shouldn’t (or fail when they shouldn’t). For crashes and segfaults, you get the line it crashed on, which is a decent enough starting point for some gdb fun.
This time, there weren’t really any exciting things to write about: the failures were minor oversights or misunderstandings that were fixed nearly instantly, giving us this in the end:
1
2
3
4
5
6
7
8
DONE!
Test run totals:
Passed: 694/18816 (3.7%)
Failed: 0/18816 (0.0%)
Not supported: 18122/18816 (96.3%)
Warnings: 0/18816 (0.0%)
Waived: 0/18816 (0.0%)
The number of unsupported formats being high is normal, there is an insane amount of obscure and niche formats out there and it’s normal not to cover everything. However, it still felt a bit higher than expected, and not just that, but while doing a simple refactor a peculiar segfault occurred, which warranted further investigation…
Pipes and Plumbing: It’s-a-me, Mar-..Mohamed
Over time, more effort has gone into unifying some common code in the Mesa stack that all drivers would use and go through, rather than each driver implementing its own thing for the exact same thing. One such is an intermediate format called “Pipe Format” that as its name suggests, acts as a pipe or intermediate stop between API color formats and the actual hardware color formats the hardware exposes. Not just that, but Pipe Format has its own helpers as well for manipulating the formats and extracting any information you could want. Alongside all that are some common Vulkan helpers that help with obtaining data for multi-planar formats such as the number of planes, or the individual format of a plane. Naturally, given the existence of the Pipe Format infrastructure, these helpers are directly wired into it, and work by converting the VK_FORMAT they take in to their equivalent PIPE_FORMAT, then calling in the Pipe Format helpers. More can be seen here, which has the common Vulkan helpers as well as the VK_FORMAT <-> PIPE_FORMAT mapping.
However, the multi-plane format representation in Pipe Format isn’t exactly 1:1 with VK_FORMAT multi-plane, and there are a few missing formats. But! They weren’t truly unsupported; the individual plane formats were supported, but a format consisting of multiple planes of these formats wasn’t. At the same time, there is another piece of common Mesa code, exclusive to Vulkan formats, called ycbcr_info, that maps much better to YCbCr VK_FORMAT formats; it represents the formats as an object containing the number of planes, the format of each plane, as well as the type of plane (i.e., luminance/chroma). For instance, here’s VK_FORMAT_G8_B8_R8_3PLANE_420_UNORM:
1
2
3
4
5
ycbcr_fmt(VK_FORMAT_G8_B8_R8_3PLANE_420_UNORM, 3,
y_plane(VK_FORMAT_R8_UNORM, YCBCR_SWIZ(G, ZERO, ZERO, ZERO), 1, 1),
c_plane(VK_FORMAT_R8_UNORM, YCBCR_SWIZ(B, ZERO, ZERO, ZERO), 2, 2),
c_plane(VK_FORMAT_R8_UNORM, YCBCR_SWIZ(R, ZERO, ZERO, ZERO), 2, 2)),
Given all this, there were two problems: firstly, the Vulkan helpers in question bypass ycbcr_info and go through Pipe Format, so any formats supported in the former but not the latter are reported as unsupported. Furthermore, there were a few missing formats from ycbcr_info, mainly 10 and 12-bit ones (which Pipe Format support, at least for 10-bit). The first problem led to the pretty high “Not Supported” count in the previous section, while the second problem led to a segfault when I (pop-quiz: why?) reworked a piece of code to rely on ycbcr_info over the common Vulkan helpers that go through Pipe Format, which is what led me on this chase in the first place.
Armed with this knowledge, the solution was rather simple: firstly, we’d add in support for 10 and 12-bit YCbCr formats into ycbcr_info, which is as simple as just filling their structures, and then after that we’d rewire things a bit (plumbing, if you will), and make it so that the Vulkan helpers that deal with multi-plane (or anything YCbCr, really) goes through ycbcr_info if the format is YCbCr, and through Pipe Format for the rest. And with all this done….
1
2
3
4
5
6
7
8
DONE!
Test run totals:
Passed: 1646/18816 (8.7%)
Failed: 0/18816 (0.0%)
Not supported: 17170/18816 (91.3%)
Warnings: 0/18816 (0.0%)
Waived: 0/18816 (0.0%)
\o/
Conclusion
Welp, that’s a wrap! Fairly lighter than the previous post, but to be fair, the work described here is far simpler. To put things in perspective, the vast majority of this was implemented in a single day; multiplane support took multiple weeks. Given the simplicity of the work, I tried to focus more on what goes into enabling it and everything around it to give a bit more perspective and detail, so hopefully even if there’s nothing really exciting, it was still an enjoyable and insightful read.
Apologies for disappearing since the last article. I graduate next year, and some graduation project stuff (sponsorship opportunities, professor mentorships) suddenly came up, and right afterwards I fell ill so I couldn’t really write much. The project itself was done and merged for quite a while now; just two more articles remain, as well as the final report. The final report has a hard deadline this Sunday, and I doubt I’ll be able to finish the remaining two articles in time, so I’ll likely post the final report later today, with the remaining articles to follow up very soon. Sorry about the awkwardness of this. In any case, thank you kindly for reading so far, hope it was a good read, and until next time! o/