Optimize "to_world" in envmap and directional emitters for automatic HDR and sunlight placement in a VFX production

[ultra-sonic]

Hi there,

First off, I just want to say I’m a huge fan of your work on Mitsuba3—it’s absolutely phenomenal! My name is Oli, and I’m the Head of 3D at RISE. When I came across Mitsuba3, my immediate thought was, "Wow, let's optimize the 'to_world' parameter for envmaps and directional lights!"

Here at RISE, we often receive HDRIs from set, accompanied by shot-specific image sequences with gray and chrome balls. Automating the "to_world" parameter would be a game-changer for processing our refBall+HDRI packages efficiently.

After experimenting with Mitsuba3, I noticed that the "to_world" parameter isn’t currently differentiable. I was curious—are there plans to make this differentiable in the future? Or is there a specific reason why this hasn’t been implemented yet?

Thank you so much for all your amazing work on Mitsuba3. I’d love to hear your thoughts on this, and I’m happy to provide any additional context if needed!

Best regards,
Oli

[ultra-sonic]

i forgot: we have the correct camera position and we (roughly) know the position and size of the greyball.