Access to SimState.rgb failed.

[RunweiSitu]

When I access the image of a viewpoint via:
state = sim.getState()
state.rgb

The following error occurs:
TypeError: Unable to convert function return value to a Python type! The signature was
(self: MatterSim.SimState) -> object

How can we access the image of a viewpoint?

[Valaybundele]

Hi @peteanderson80,

I am also facing the same issue. Please let us know how to visualise the images.

Thanks

[Valaybundele]

Ok, so I found a solution. It's also mentioned in the readme, in the section which details the output of the simulator. To get the np array for a view, do np.array(state.rgb, copy=False). If you get an array of zeros in the output, make sure that you have downloaded the camera parameters along with the skybox images.

[zhangjb416]

Ok, so I found a solution. It's also mentioned in the readme, in the section which details the output of the simulator. To get the np array for a view, do np.array(state.rgb, copy=False). If you get an array of zeros in the output, make sure that you have downloaded the camera parameters along with the skybox images.

Hi! I met the same problem that np.array(state.rgb, copy=False) return an array of zeros in the output.

But I have downloaded the camera parameters (matterport_camera_intrinsics, matterport_camera_poses, undistorted_camera_parameters) and skybox images. These data are put in a scan folder (e.g., ./Matterport3D/17DRP5sb8fy/matterport_skybox_images). Do you have any idea why the output is still an array of zeros?

Thanks a lot!

[LinqingZhong]

Ok, so I found a solution. It's also mentioned in the readme, in the section which details the output of the simulator. To get the np array for a view, do np.array(state.rgb, copy=False). If you get an array of zeros in the output, make sure that you have downloaded the camera parameters along with the skybox images.

Hi! I met the same problem that np.array(state.rgb, copy=False) return an array of zeros in the output.

But I have downloaded the camera parameters (matterport_camera_intrinsics, matterport_camera_poses, undistorted_camera_parameters) and skybox images. These data are put in a scan folder (e.g., ./Matterport3D/17DRP5sb8fy/matterport_skybox_images). Do you have any idea why the output is still an array of zeros?

Thanks a lot!

same issue . May I ask have you solved this problem?

[LambdaGuard]

The key problem is to initialize the simulator with render=True or it will not load any rgb images. e.g.

sim = MatterSim.Simulator()
sim.setRenderingEnabled(True)
sim.setDiscretizedViewingAngles(True)
sim.setBatchSize(1)
sim.setCameraResolution(640,480)
sim.setCameraVFOV(math.radians(60))
sim.initialize()

However this will raise an error when you do not have any attached displays. In this case you can use the xvfb to create a virtual display:

sudo apt install xvfb
Xvfb :99 -screen 0 1024x768x24 &
export DISPLAY=:99

Then the getState() will contain the RGB image. NOTICE that you should run

./scripts/downsize_skybox.py

to downsize the images forthey are the files the simulator will load.