How to Set Up a 32-Channel BRDF in Spectral Rendering

[free7187]

Thanks to the discussion#1309 , I can now attempt to edit spectral multi-channel rendering in Python.

However, the working mode for spectral channels is not quite what I expected. To test whether I could edit individual channels of the spectral BRDF, I ran a simple test with the following code:

import sys
sys.path.insert(0, "./mitsuba3/build/Release/python")
import mitsuba as mi
import drjit as dr 
import torch
import torch.nn.functional as F
import torch.nn as nn
import torch.optim as optim
import numpy as np
import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
mi.set_variant('cuda_ad_spectral')
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
class real_data_integrator(mi.SamplingIntegrator):
    def __init__(self, props):
        super().__init__(props)
    def sample(self, 
               scene: mi.Scene, 
               sampler:mi.Sampler, 
               ray: mi.RayDifferential3f, 
               medium: mi.Medium = None,
               active: bool = True) -> tuple[mi.Color3f, bool, list[float]]:  
        L = mi.Spectrum(0.0)
        si = scene.ray_intersect(ray, active)        
        valid_ray = active & si.is_valid()
        active &= si.is_valid()
        ds, em_weight = scene.sample_emitter_direction(
            si, sampler.next_2d(), True, active
        )
        active &= ds.pdf != 0.0
        brdf = mi.Spectrum(0.0)
        new_brdf= mi.Spectrum(self.set_value(brdf))
        L += new_brdf *em_weight
        return dr.select(True, L, 0.0), valid_ray, []
    
    @dr.wrap(source='drjit', target='torch')
    def set_value(self,brdf):
        new_brdf=brdf
        new_brdf[0:10,:]=1
        # new_brdf[10:20,:]=1
        # new_brdf[20:30,:]=1
        # new_brdf[:,:]=1
        # new_brdf[20:21,:]=1
        return new_brdf

mi.register_integrator('My_integrator', lambda props: real_data_integrator(props))

xml_path= "spectrum_test.xml"
scene = mi.load_file(xml_path)
with dr.suspend_grad():
    with torch.no_grad():
        image=mi.render(scene)
print("rendering done")
mi.util.write_bitmap("test_spectrum0.png", image)
mi.util.write_bitmap("test_spectrum0.exr", image)

And here is the XML scene setup:

<scene version="3.0.0">
    <default name="spp" value="512"/>
    <default name="res" value="512"/>
    <integrator type="My_integrator"/>

    <sensor type="perspective" id="sensor">
        <string name="fov_axis" value="smaller"/>
        <float name="near_clip" value="0.001"/>
        <float name="far_clip" value="100.0"/>
        <float name="focus_distance" value="1000"/>
        <float name="fov" value="40"/>
        <transform name="to_world">
            <lookat origin="0,  0,  3.6"
                    target="0,  0,  0"
                    up    ="0,  1,  0"/>
        </transform>
        <sampler type="independent">
            <integer name="sample_count" value="$spp"/>
        </sampler>
        <film type="hdrfilm">
            <integer name="width"  value="$res"/>
            <integer name="height" value="$res"/>
            <rfilter type="tent"/>
            <string name="pixel_format" value="rgb"/>
            <string name="component_format" value="float32"/>
        </film>
    </sensor>
<emitter type="directional">
    <vector name="direction" value="0.0, 0.0, -1.0"/>
    <rgb name="irradiance" value="1.0"/>
</emitter>
    <shape type="obj">
		<string name="filename" value="cube.obj" />
		<transform name="to_world">
            <scale value="1"/>
            <translate x="0" y="0" z="0"/>
        </transform>
        <bsdf type="diffuse">
            <rgb name="reflectance" value="1.0"/>
        </bsdf>
	</shape>
</scene>

I initially assumed that with the 32-channel setup, setting channels [0:10], [10:20], and [20:30] of new_brdf to 1 (with other channels set to 0) would result in different colors for the render output. However, the final render always appears gray:

I also tried setting all BRDF channels to 1, as well as setting only one specific channel to 1. The render results are as follows:


After analyzing the RGB ratios for all pixels in these three images, I noticed that they all tend towards (1:1:1). It appears that changing channel values only affects the final output’s brightness, which is quite different from what I had anticipated. I originally expected, for example, that setting channels [10:20] to 1 and others to 0 would result in a green color.

Am I missing something? How should I properly edit spectral BRDF channels to achieve the intended effect?

[merlinND]

Hello @free7187,

Did you check the values of ray.wavelengths? I don't remember the details, but wavelengths may not be sampled in increasing order, and so setting some channels to a certain values could affect wavelengths that you don't expect.
Note also that by default, each ray samples different wavelengths.

PS: when debugging things like this, I would recommend removing torch usage and finding the simplest possible test case, because the more components you add, the more confusing it can be to interpret the results.

[free7187]

It seems there is a misunderstanding here. I did not forcefully modify the wavelengths; I simply fixed the variable wavelength_sample to 0. This variable is only called here, and the code mentions its purpose as

'A uniformly distributed 1D value that is used to sample the spectral dimension of the sensitivity profile.'

Therefore, I only modified that variable, and I believe it should not lead to incorrect generation of wav_weight or other issues.

[merlinND]

In Mitsuba3, I only modified the code above. Additionally, I set up SRFs, and now I have forcibly set ray.wavelengths under spectral mode rendering to 400nm, 410nm, ..., 700nm, 710nm, totaling 32 channels

This is what I was referring to. You mentioned forcibly setting the wavelengths.

[free7187]

Sorry, my previous statement was misleading. The SRF I imported is:

400.000000 1.000000
410.000000 1.000000
420.000000 1.000000
430.000000 1.000000
440.000000 1.000000
...
710.000000 1.000000
720.000000 1.000000

With my previous modifications, by fixing wavelength_sample to 0, the rays generated by the sensor start at 400nm, and due to the 32-channel setup, the wavelengths will be spaced 10nm apart. This way, I fixed the sampled wavelengths to be "400nm, 410nm, ..., 700nm, 710nm, totaling 32 channels." However, I also tried using this SRF setup in rendering with Mitsuba 3.5.0 without any modifications, and it didn't have any effect, so I believe the issue isn't here."

[free7187]

Is it only volpathmis that's affected, not any other integrator? (specifically path with some simple shapes & BSDFs)

I have tested the direct and path Integrators with spectral BSDF. I also tested the volpath and volpathmis using the null BSDF, but with spectral medium. The test results indicate that the issue only appears in volpath and volpathmis that include a medium.

However, I suddenly realized one point: my goal is simply to render a spectral channel volume rendering result as a reference. Initially, I was concerned that the unmodified Mitsuba used only 4 channels in spectral rendering, which might lead to deviations from expected results.
Yet, the observed results show that the modified 32-channel output matches exactly with the original 4-channel direct with spectral BSDF rendering. Therefore, perhaps I can use the original volume rendering method to obtain my reference.

[free7187]

Additionally, I've found a problem. Even using same rendering parameters, the color results from volpath and volpathmis differ under the unmodified version of Mitsuba in scalar_spectral mode.
volpath:

volapthmis:

The official documentation mentions volpath "This integrator does not implement good sampling strategies to render participating media with a spectrally varying extinction coefficient." Bur I understood "does not implement good sampling" to mean that rendering result has more noise and requires more spp. How is the color discrepancy generated?

Here is my XML configuration:

<scene version="3.0.0">
	<default name="integrator" value="volpath" />
	<default name="spp" value="50000" />
	<!-- <default name="integrator" value="volpathmis" />
	<default name="spp" value="1000" /> -->
	<default name="res" value="512" />
	<default name="max_depth" value="32" />
	<integrator type="$integrator">
		<integer name="max_depth" value="$max_depth" />
	</integrator>
    
    <sensor type="perspective" id="sensor">
        <string name="fov_axis" value="smaller"/>
        <float name="near_clip" value="0.001"/>
        <float name="far_clip" value="100.0"/>
        <float name="focus_distance" value="1000"/>
        <float name="fov" value="40"/>
        <spectrum name="srf" filename="test.spd"/>
        <transform name="to_world">
            <lookat origin="0,  0,  3.6"
                    target="0,  0,  0"
                    up    ="0,  1,  0"/>
        </transform>

        <sampler type="independent">
            <integer name="sample_count" value="$spp"/>
        </sampler>
        <film type="hdrfilm">
            <integer name="width"  value="$res"/>
            <integer name="height" value="$res"/>
            <rfilter type="tent"/>
            <string name="pixel_format" value="rgb"/>
            <string name="component_format" value="float32"/>
        </film>
    </sensor>
    
<emitter type="directional">
    <vector name="direction" value="0.0, 0.0, -1.0"/>
    <rgb name="irradiance" value="2.0"/>
</emitter>
    <!-- Shapes -->
    <shape type="obj">
		<string name="filename" value="cube.obj" />
		<transform name="to_world">
            <scale value="1"/>
            <translate x="0" y="0" z="0"/>
        </transform>
        <bsdf type="null">
        </bsdf>
		<medium type="homogeneous" name="interior">
            <spectrum name="sigma_t" value="400:0.1346,410:0.1254,420:0.1173,430:0.11,440:0.1035,450:0.09777,460:0.09258,470:0.08792,480:0.08372,490:0.07993,500:0.07649,510:0.07338,520:0.07054,530:0.06795,540:0.06559,550:0.06343,560:0.06145,570:0.05962,580:0.05795,590:0.0564,600:0.05497,610:0.05365,620:0.05243,630:0.05129,640:0.05023,650:0.04925,660:0.04833,670:0.04747,680:0.04667,690:0.04592,700:0.04522,710:0.04455" />
			<spectrum name="albedo" value="400:0.9316,410:0.9337,420:0.9358,430:0.9379,440:0.94,450:0.942,460:0.9439,470:0.9458,480:0.9477,490:0.9495,500:0.9512,510:0.9529,520:0.9545,530:0.956,540:0.9575,550:0.9589,560:0.9603,570:0.9616,580:0.9628,590:0.964,600:0.9652,610:0.9662,620:0.9673,630:0.9683,640:0.9692,650:0.9701,660:0.971,670:0.9718,680:0.9726,690:0.9733,700:0.974,710:0.9747" />
			<float name="scale" value="10000"/>
		</medium>
	</shape>
    <shape type="obj">
		<string name="filename" value="inner_cube.obj" />
		<transform name="to_world">
            <scale value="1"/>
            <translate x="0" y="0" z="0"/>
        </transform>
        <bsdf type="null">
        </bsdf>
		<medium type="homogeneous" name="interior">
            <spectrum name="sigma_t" value="400:0.07184,410:0.07197,420:0.06901,430:0.06162,440:0.05486,450:0.04921,460:0.04582,470:0.04332,480:0.04116,490:0.03931,500:0.03764,510:0.03617,520:0.03494,530:0.03408,540:0.03327,550:0.03209,560:0.03094,570:0.03026,580:0.02953,590:0.02802,600:0.02704,610:0.02634,620:0.02574,630:0.02519,640:0.02468,650:0.02421,660:0.02378,670:0.02337,680:0.02299,690:0.02264,700:0.0223,710:0.02199" />
			<spectrum name="albedo" value="400:0.8726,410:0.8134,420:0.795,430:0.8371,440:0.887,450:0.9357,460:0.9536,470:0.9599,480:0.9638,490:0.9654,500:0.9666,510:0.9666,520:0.9634,530:0.9532,540:0.9437,550:0.9477,560:0.9534,570:0.9472,580:0.9447,590:0.9701,600:0.9813,610:0.984,620:0.9852,630:0.986,640:0.9864,650:0.9867,660:0.9869,670:0.9871,680:0.9872,690:0.9873,700:0.9873,710:0.9873" />
			<float name="scale" value="10000"/>
		</medium>
	</shape>
</scene>

[wjakob]

Hi @free7187 (cc @merlinND),

what you are trying to do is not supported by Mitsuba 3. Mitsuba 3 performs spectral rendering using a continuous Monte Carlo sampling of the spectral domain. Increasing the number of spectral channels will accelerate the convergence of this Monte Carlo procedure at the cost of higher register usage and computation. It's a subtle tradeoff between the cost of the computation and the convergence of the Monte Carlo integral. A value of 4 spectral samples is chosen as default, and you will likely not get much of a performance benefit by going higher than that.

In contrast, discrete rendering using a specifically chosen set of wavelengths is not supported. You may be able to hack Mitsuba to do this, but we cannot provide support for such a feature here -- you will be on your own.