Question on capturing RGB(YUV) frames on client and Monado version ALVR

[yingtian991221]

Hi,

Similar to Jia-Wei's problem (#1311), I'm currently doing researching on capturing RGB videos and real-time processing and then display on screen. My current solution is to capture pixels when the pointer is passed to EGL environment (where textures can be transformed to RGB data) and my idea was to do processing then and display RGB frames back to GL. Are there any previous stages I can extract the textures to RGB pixels? I believe H.264 actually transfers compressed YUV pixels, so the textures are some form of YUV too, how can I extract them?

I have another question as I'm now on Quest 2 platform, but I find it actually not support GPU/DSP for ml (I run snapdragon NPE and found gpu and dsp networks are not supported) so I'm considering changing to a Snapdragon 865XR development board, which has Android 10 and very likely to support Mondao, so how can I access Mondao version ALVR for any test?

Best,
Yingtian

[zmerp]

Hi, what is your goal precisely? Do you think that what you want can be done on the server instead?

ALVR does not currently support generic Android devices or Monado, some people report not working at all, some say that the aspect ratio is wrong. I don't have a suitable android device to test.

[yingtian991221]

I have compared a few games, like beat saber and doom3, and found that ALVR usually has better fps (espacially Doom3, ALVR keeps 72 fps while Doom3 quest by Dr. Beef maintain 30-45fps) but tend to a decended visual quality, so I think maybe transferring a lower quality image and then super-resolution is worth trying, or is there such functionality in progress already?

[zmerp]

There have been discussions about using superresolution, but nothing concrete. The XR2 probably is too underpowered

[yingtian991221]

Some previous works combined super-resolution and bicubic interpolation may compensate for realtime super-resolution, but on Quest2 I even doubt they support running models on GPU?

[yingtian991221]

Works like NEMO from https://dl.acm.org/doi/10.1145/3372224.3419185