Implement motion vectors in compatibility renderer

[devloglogan]

This PR implements motion vectors in the compatibility renderer, primarily for use with the XR_FB_space_warp OpenXR extension. For testing purposes, this PR currently implements this extension as well. Eventually this logic will be handled via GDExtension in godot_openxr_vendors. This PR will remain as a draft until that's the case.

Edit: Space warp functionality has been relocated to GodotVR/godot_openxr_vendors#222.

Technical details

I've tried to emulate how motion vectors are implemented in the Forward+ renderer where possible, but it's worth noting that space warp requires 3D motion vectors. Whereas the Forward+ renderer is rendering 2D motion vectors for TAA.

Motion vectors are rendered in a new, separate render pass. Why this was done:

• Since motion vectors can be rendered at a much smaller resolution this extra pass should be quite fast (e.g. the Quest 3 recommends a resolution of around 420x420).
• This will make it easier to allow developers to include or exclude specific objects from motion vectors, which can be helpful in minimizing the artifacts from space warp. (Note: this isn’t part of the current PR, but something planned for the future.)
• Combining motion vectors with the color pass would hinder future compatibility between Application Space Warp and the XR_META_recommended_layer_resolution OpenXR extension, since all attachments would then have to be of the same resolution.

The vertex shader code in scene.glsl has been placed in a new vertex_shader() function. When motion vectors are used, it will run the vertex shader on both previous and current vertex data. The scene state data and multiview data have been duplicated to hold the previous frame’s data. The vertex shader outputs both the previous and current clip positions, which are used to calculate motion vectors in the fragment shader.

For skeletons, GLES::MeshInstance::Surface now stores two vertex buffers, generating the second one on demand when motion vectors are enabled. The VBOs will alternate between current/previous, and the previous VBO is used to populate new scene.glsl vertex attribs at locations 16/17.

Similarly, for multimeshes/particles, GLES::MultiMesh now also stores two vertex buffers with the secondary one generated on demand. New scene.glsl vertex attribs (location 18 to 21) store the previous data, and the setup for both the current/previous vertex attribs is now done in a new GLES3::MeshStorage::multimesh_vertex_attrib_setup() function.

Issues that need to be addressed

There are a couple things that need to be ironed out that I would really appreciate some input on from those who have a bit more rendering know how.

• This implementation of motion vectors requires the GPU to have a max vertex attribute value of at least 22. This requirement is met on Quest hardware (or any other headset which uses the Qualcomm XR-series chips), and we check it before running the motion vector pass. But when opening the editor on desktop some errors might be thrown from ShaderGLES3::_initialize_version(), which attempts to compile all shader variants at startup, including the motion vector variant of scene.glsl. What is the best way to prevent it from compiling this variant if there aren’t enough vertex attributes?

• In non-one-shot particle systems, it's possible for a particle instance to go from the end of its lifetime one frame to the beginning of its lifetime the next frame. At the moment this causes an undesired result on the motion vector texture (we should draw nothing for the particle on these frames). Is there a way we can determine if this is the case in scene.glsl?

• As mentioned above the vertex shader in scene.glsl has moved into a new vertex_shader() function. For multiview data, I'm currently passing in a boolean value indicating whether to use multiview_data_block.prev_data or multiview_data_block.data. I'd rather just pass in the MultiviewData structs themselves rather than the boolean, similar to how the SceneData structs are passed in. However, doing this seems to cause anything using multiview data to just not be rendered. What I have in the PR works, so maybe it's not a big deal, but if anybody has ideas on how to fix this I'd appreciate it!

UPDATED (2024-10-28): I believe all of the areas of concern have been addressed. While technically the second point regarding particle systems still exists, I don't think this is something that needs to be addressed in this PR. Beyond just particles, there should be a way to exclude objects from being drawn to motion vectors (Ex: when an object is teleporting rather than moving smoothly). This is something the current Vulkan motion vectors implementation could benefit from. I think a fix for this warrants its own PR.

Contributed with ❤️ by W4Games

[clayjohn]

This implementation of motion vectors requires the GPU to have a max vertex attribute value of at least 22. This requirement is met on Quest hardware (or any other headset which uses the Qualcomm XR-series chips), and we check it before running the motion vector pass. But when opening the editor on desktop some errors might be thrown from ShaderGLES3::_initialize_version(), which attempts to compile all shader variants at startup, including the motion vector variant of scene.glsl. What is the best way to prevent it from compiling this variant if there aren’t enough vertex attributes?

Yes, motion vectors should be a specialization for now. Then the version will only be compiled when it is actually used.

In non-one-shot particle systems, it's possible for a particle instance to go from the end of its lifetime one frame to the beginning of its lifetime the next frame. At the moment this causes an undesired result on the motion vector texture (we should draw nothing for the particle on these frames). Is there a way we can determine if this is the case in scene.glsl?

Not from scene.glsl, no. We will have to do something in the actual particle system. I think that we handled this case already in the RD renderers. Most likely we will need to duplicate the particle buffer to effectively clear the history

As mentioned above the vertex shader in scene.glsl has moved into a new vertex_shader() function. For multiview data, I'm currently passing in a boolean value indicating whether to use multiview_data_block.prev_data or multiview_data_block.data. I'd rather just pass in the MultiviewData structs themselves rather than the boolean, similar to how the SceneData structs are passed in. However, doing this seems to cause anything using multiview data to just not be rendered. What I have in the PR works, so maybe it's not a big deal, but if anybody has ideas on how to fix this I'd appreciate it!

I can't think of any reason why one struct would work but not another. We should try to get it working though as the bool will likely create a dynamic branch which we really don't want

[dsnopek]

Thanks! This is looking really great to me :-)

I'm going to test the latest version soon, but in the meantime, I have some notes on the public APIs this adds. Some of this will need discussion with the rest of the XR team.

[dsnopek]

Doing some testing with AppSW via GodotVR/godot_openxr_vendors#222, I'm getting this message spammed to logs:

I don't see why we should be seeing this unhandled type, I think this is meant to be something that the application logs via OpenGL? And I don't think Godot is doing it. Maybe it's something from the OpenXR runtime?

But just this getting sent to the log so often is going to have a negative affect on performance. One option could be updating _gl_debug_print() to handle this type? Or, I guess figure out what is doing this logging, and somehow stop it from doing that?

These appear to be the types we're not handling:

#define _EXT_DEBUG_TYPE_MARKER_ARB              0x8268
#define _EXT_DEBUG_TYPE_PUSH_GROUP_ARB          0x8269
#define _EXT_DEBUG_TYPE_POP_GROUP_ARB           0x826A

[devloglogan]

@dsnopek it's of type "Marker" and severity "Notification", I noticed this recently and added the code to make it actually share the log and this is what it shared:

ERROR: GL ERROR: Source: Third Party Type: Marker    ID: 1   Severity: Notification  Message: vr-marker,frame_end,type,application,frame_num,175
at: _gl_debug_print (drivers\gles3\rasterizer_gles3.cpp:188)

Edit: It logged this as well

ERROR: GL ERROR: Source: Third Party Type: Marker    ID: 1   Severity: Notification  Message: VR Layer 0, SubImage Rect L0.0/0.0/1680.0/1760.0 R0.0/0.0/1680.0/1760.0, ColorScale 1.0/1.0/1.0/1.0
at: _gl_debug_print (drivers\gles3\rasterizer_gles3.cpp:188)

I can add the code to this PR so that its actually logged like above, but yes I'm thinking I'll have to figure out whats causing this and try to stop it. Since it's Marker/Notification I'm assuming what causes this is not actually a problem, but the log itself would be an issue.

[BastiaanOlij]

@devloglogan @dsnopek I've seen those errors for some time now, even before we added support for OpenXR debug logging (which is turned off by default anyway).
We don't seem to have a way to control the level of logging coming from the OpenGL message callback

[devloglogan]

I'm opening this PR since the XR_FB_space_warp code has been removed and it's in a now arguably merge-able state. :)

[clayjohn]

From a general architectural standpoint, I think this makes sense. Before merging I would like to see some performance testing on mobile devices as I suspect that the shader changes might cause a slight regression in performance for the non motion vector case.

The API for passing in the external motion vector buffer is very awkward in light of the fact that the old override API already has a slot for the depth buffer and the velocity buffer. I don't think overlapping with the old API is great as it is layering on technical debt. I suspect we can merge this with the existing override API to streamline things and avoid creating a new, conflicting API surface.

The one outstanding question I have is how this will work in the Mobile renderer. I suspect that doing a pre-pass is fine for two reasons:

1. We have to have it anyway if we are going to support space warp.
2. We can likely set it up so the vertex workload of the main pass fully overlaps with the fragment workload in the motion vector pass. This would minimize the performance impact of the pass. If other consumers of motion vectors are okay with a low res pass, then it could be really good.

[dsnopek]

@clayjohn:

The API for passing in the external motion vector buffer is very awkward in light of the fact that the old override API already has a slot for the depth buffer and the velocity buffer. I don't think overlapping with the old API is great as it is layering on technical debt. I suspect we can merge this with the existing override API to streamline things and avoid creating a new, conflicting API surface.

The thing is, it's a different depth buffer (at a different size).

On the pre-existing one, the color and depth texture there are for the color pass:

render_target_set_override(RID p_render_target, RID p_color_texture, RID p_depth_texture, RID p_velocity_texture)

On the new one, they are for the motion vector pass:

void render_target_set_motion_vectors_target(RID p_render_target, RID p_motion_vectors_texture, RID p_depth_texture, Size2i p_size, int p_view_count)

And the p_depth_texture for each are two completely different textures. So, to me, it makes sense to have different function for overriding stuff for the two different passes. If we used the original p_velocity_texture on the old function, but added a new function for the 2nd depth texture and the size, that could seem weird because the p_velocity_texture doesn't go with the color pass.

But now that I'm looking at this again, we might not need to pass p_view_count - we probably have that somewhere else already.

[clayjohn]

@dsnopek I would just add a parameter to the existing function for the velocity depth texture.

[dsnopek]

@dsnopek I would just add a parameter to the existing function for the velocity depth texture.

Ah, ok, that makes sense! Then we can just have a new method for setting the motion vector target size.

[devloglogan]

@dsnopek @BastiaanOlij I've added _EXT_DEBUG_TYPE_MARKER_ARB to the ignored opengl types to suppress that error log.

@clayjohn I've made changes to address your suggestions, thanks for the review!

[dsnopek]

I have no more notes. :-) This is looking great to me!

[devloglogan]

@AThousandShips thanks so much for the review! I've applied all your suggestions :)

[dsnopek]

This'll need to be rebased now that PR #98831 was merged (sorry :-))

[devloglogan]

@dsnopek No worries, rebased. :)