Using matplotlib with python bindings

[sp7412]

How do I use matplotlib with displayFrame? Also, is the dataset for visp/modules/python/examples/synthetic-data-mbt.py available for download?

I = visp.core.ImageGray()
visp.io.ImageIo.read(I, './boom_test_4_8_2024_undistorted/0000_LWIRU2.png')
# Display.displayFrame(I, T_cam_model_init,cam,3,)
# tmp = Display.getImage
# visp.core.Display.displayText(I, 0, 0, 'Click to initialize tracking', visp.core.Color.red)
# visp.core.Display.flush(I)
# tracker.display(visp.core.ImageGray(I),T_cam_model_init)

[fspindle]

The dataset is available in tutorial/tracking/model-based/generic-rgbd-blender.

$ python3 synthetic-data-mbt.py --data-root $VISP_WS/visp/tutorial/tracking/model-based/generic-rgbd-blender

Can you precise your question concerning matplotlib ?

PS: @SamFlt for info

[sp7412]

I would like to reproduce this example in python: https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-mb-generic-tracker-stereo_8cpp-example.html

For the plots, I would like to use matplotlib rather than opencv (because I'm using jupyter notebook).

I was able to compile the python bindings on ubuntu but it doesn't seem like the visualization functions like:

Display.displayFrame(I, T_cam_model_init,cam,3,)

actually return the updated image.

So, basic question is: how do I use the python bindings with matplotlib figures? And further, do you have any jupyter notebook examples?

[SamFlt]

Hi,

Indeed, ViSP displays do not update the image directly: they store a buffer which contains the displayed image as well as the render overlay (the pose frame in your example)

To combine ViSP plots with matplotlib, you can use something like:

from visp.core import Display, ImageRGBa, RGBa, HomogeneousMatrix, CameraParameters, Color
import numpy as np
try:
  from visp.gui import DisplayX
  DisplayCls = DisplayX
except ImportError:
  from visp.gui import DisplayOpenCV
  DisplayCls = DisplayOpenCV

import matplotlib.pyplot as plt

# black image
I = ImageRGBa(240, 320, RGBa(0, 0, 0, 255))
d = DisplayCls()
d.init(I, win_x=0, win_y=0, win_title='ViSP display')
cTo = HomogeneousMatrix(0, 0.1, 0.7, np.radians(45), np.radians(20), 0)
cam = CameraParameters(300, 300, 160,  120)

# First step: perform VisP related plotting
Display.display(I)
Display.displayFrame(I, cTo, cam, size=0.1, color=Color.none)

# Fetch the result: Image with the frame displayed
displayI = ImageRGBa()
Display.getImage(I, displayI)

# Display in matplotlib and additional plots
fig = plt.figure()
plt.title('Matplotlib image')
plt.imshow(displayI) # ViSP image is interpreted as a Python buffer: matplotlib can automatically read it
plt.scatter(np.random.uniform(0, 320, 50), np.random.uniform(0, 240, 50))
plt.show()

The method of interest here is Display.getImage, which should be called to get the overlay after you've rendered the pose frame in ViSP.
You can directly give the ViSP image to matplotlib and it will be correctly interpreted. It should also be true for other types, such as matrices, and vectors.

Thanks for your question and do not hesitate to @ me if you have any feedback or questions!
Cheers

[sp7412]

This is helpful and I'm able to see the image I loaded but not the overlaid frame.

I = ImageRGBa()
visp.io.ImageIo.read(I, 'image.png')

cTo = HomogeneousMatrix(0, 0.1, 0.7, np.radians(45), np.radians(20), 0)
cam = CameraParameters(300, 300, 160,  120)

# First step: perform VisP related plotting
Display.display(I)
Display.displayFrame(I, cTo, cam, size=0.1, color=Color.none)

# Fetch the result: Image with the frame displayed
displayI = ImageRGBa()
Display.getImage(I, displayI)

# Display in matplotlib and additional plots
fig = plt.figure()
plt.title('Matplotlib image')
plt.imshow(displayI)
plt.show()

[SamFlt]

I believe this is because you are not creating a Display object. In the example i gave above you have these lines:

d = DisplayCls() # DisplayX, DisplayOpenCV or other
d.init(I, win_x=0, win_y=0, win_title='ViSP display')

Without these lines, no display overlay will be created and calls to Display will have no effect.

Sam

[sp7412]

I did try rerunning the full code snippet you provided in a vs code jupyter notebook instance and the figure that pops up is blank.