Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basler Camera with Pylon SDK Errors #16

Open
lumyus opened this issue Apr 20, 2021 · 18 comments
Open

Basler Camera with Pylon SDK Errors #16

lumyus opened this issue Apr 20, 2021 · 18 comments

Comments

@lumyus
Copy link

lumyus commented Apr 20, 2021

Hey,

I built Fictrac with Cmake and the following command: cmake -D CMAKE_TOOLCHAIN_FILE=/home/vcpkg/scripts/buildsystems/vcpkg.cmake -D BASLER_USB3=ON -D BASLER_DIR="\opt\pylon5" ..
I am using PylonSDK 5 (I also tried PylonSDK 6 but the build would not work) and we are running everything on Ubuntu.

This is the error I get when running Fictrac:

Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
[ INFO:0] global /home/nely/opencv/modules/videoio/src/videoio_registry.cpp (187) VideoBackendRegistry VIDEOIO: Enabled backends(7, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); FIREWIRE(940)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (248) getPluginCandidates VideoIO pluigin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (256) getPluginCandidates - /usr/local/lib: 0
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (259) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ INFO:0] global /home/nely/opencv/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: %08d @ 40022761
Could not interpret source type (40022761)!
Error! Could not open input frame source (40022761)!
Error loading parameters from specified config file (config.txt)!

I also tried to change the src_fn from 40022761 to 0 but the error stayed the same. (40022761 is the camera id in PylonViewerApp)
I also tried to build without the Pylon SDK (cmake -D CMAKE_TOOLCHAIN_FILE=/home/vcpkg/scripts/buildsystems/vcpkg.cmake) but the error stayed the same.
Somehow we are not getting any output with Fictrac, neither with the ConfigGui.
The config.txt used is the same as the one provided in the /sample folder. (Except src_fn has been changed)
Any idea on what is going wrong?

Thanks for your help!

@rjdmoore
Copy link
Owner

rjdmoore commented Apr 22, 2021

Hey, ok so camera id is probably a poor name. It's actually the USB id (nth device on bus). If you only have one usb device connected, src_fn should be 0. If you have more than one device, src_fn might be 1, 2, ... depending on device enumeration.

If you've tried setting src_fn to 0, 1, 2.. and still not getting it to work, then this suggests something else is wrong. You should make sure the camera is properly configured and streaming before starting fictrac. You should probably be also close basler viewer app prior to starting fictrac.

@lumyus
Copy link
Author

lumyus commented Apr 22, 2021

Thanks for getting back to me. We tried 0,1,2 etc but had no success.
You mentioned "properly configured and streaming": We configure the camera as always with the Basler Viewer App and then open the stream (Red to Green button) and close the Basler Viewer App subsequently. The issue remains.
Is it necessary to build Fictrac with BASLER_USB3=ON in order to use the Basler Cameras? Or could we also run without the Pylon SDK?
If the Pylon SDK is necessary, which version did you test it with (5 or 6)?

@rjdmoore
Copy link
Owner

Hmm, ok. Basler cameras are a little poorly tested unfortunately, so there could be an API change in the newer versions that's causing an issue. Basler code was originally added June 2019 I think, so that's probably the last version that was tested properly.

I will try and get hold of a basler camera to test with, but if there's API issues you should be seeing other error messages, at least in the console.

In the meantime you could try posting in the forum (Reddit) to see if anyone has more recent experience with basler and fictrac. You could also try contacting directly those that have posted basler questions in the past.

You should also try just using the basler camera without the SDK. Just build without those options. You probably won't be able to use advanced camera modes, so try setting to some default standard res/FPS/bitdepth first and see if it works! You'll still need to find which USB id your camera is using (should be zero if you have no other cameras).

@lumyus
Copy link
Author

lumyus commented Apr 22, 2021

Okey so I did the following and it still doesn't work:
I built Fictrac with no SDK specified. (Regular build command). Then I called lsusb which shows:

_**Bus 014 Device 005: ID 2676:ba02 Basler AG ace**
Bus 014 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 013 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 012 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 003: ID 046d:c062 Logitech, Inc. M-UAS144 [LS1 Laser Mouse]
Bus 001 Device 004: ID 046a:0023 Cherry GmbH CyMotion Master Linux Keyboard G230
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub_

I therefor put src_fn = 5 .
The error I get when running Fictrac is now:

_Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
VIDEOIO ERROR: V4L: can't open camera by index 5
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (248) getPluginCandidates VideoIO pluigin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (256) getPluginCandidates     - /usr/local/lib: 0
[ INFO:0] global /home/nely/opencv/modules/videoio/src/backend_plugin.cpp (259) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ INFO:0] global /home/nely/opencv/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: %01d @ 5
Could not interpret source type (5)!
Error! Could not open input frame source (5)!
Closing sphere tracker
Closing recorder..
Closing recorder.._

Does this help you to debug? I am still lost to be honest. I will reach out on Reddit to the others and thanks for the help so far!

@lumyus
Copy link
Author

lumyus commented Apr 22, 2021

Actually upon building with SDK 5 specified we get the following:

Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
Error opening capture device! Error was: Failed to open device '2676:ba02:14:4:5' for XML file download. Error: 'Device is exclusively opened by another client.'
Error! Could not open input frame source (5)!

So of course we closed the Pylon viewer to not have another client using the camera but this will result in the following error:

Error! Unable to set process priority!
Looking for config file: config.txt ..
Config file parsed (21 key/value pairs).
Opening Basler camera device: �*�p�
Error opening capture device! Error was: NULL pointer dereferenced
Error! Could not open input frame source (5)!
Closing sphere tracker

@rjdmoore
Copy link
Owner

rjdmoore commented Apr 22, 2021 via email

@lumyus
Copy link
Author

lumyus commented Apr 23, 2021

So I verified that Pylon SDK 5.2 is not working either. The errors remain.
What OpenCV version did you use? Found OpenCV: /usr/local (found version "4.1.0") is what we use.

Any other ideas on what might cause the issue? How exactly are we supposed to 'open' the camera stream? Do I have to edit anything at all in the Pylon Viewer?

@lumyus
Copy link
Author

lumyus commented Apr 23, 2021

@rjdmoore
Copy link
Owner

rjdmoore commented Apr 23, 2021 via email

@lumyus
Copy link
Author

lumyus commented Apr 23, 2021

Yes we can do that together. We have a bunch of Basler Ace at hand in our lab. How do you want to proceed?

@lumyus
Copy link
Author

lumyus commented Apr 26, 2021

I did some more testing today and it seems that first of all:
LOG("Opening Basler camera device: %s", _cam.GetDeviceInfo().GetModelName());
does not show a proper model name in the terminal but this did:

 // Print the model name of the camera.
cout << "Opening Basler camera device:" << camera.GetDeviceInfo().GetModelName() << endl;

and I get the correct camera name as output (which is already a good sign).
Now the null pointer exception I was talking about happens in this method:
_fps = getFPS();
but I saw that you didn't test this yet. (It's a TODO).
Right now I still have a frame grabbing error and I don't know how to resolve that.

I also saw that you defined "BaslerSource::BaslerSource(int index)" but never actually use the index anywhere, right?

Let me know if this helps you. Thanks again for your awesome work!

@lumyus
Copy link
Author

lumyus commented Apr 27, 2021

Config file parsed (21 key/value pairs).
Using device acA1920-150um
Getting frame rate..
Basler camera initialised (1920x1200 @ 112.499 fps)!
Key (output_fn) not found.
Key (fisheye) not found.
Key (roi_c) not found.
Warning! Re-computed sphere ROI centred at [-0.490091 -0.243093 0.837088], with radius 0.023668 rad from 4 roi_circ points.
Input sphere mask automatically generated using 2 ignore ROIs!
roi_to_cam_r: -0.2572 0.5186 0.0000
Found C2A rotational transform: [0.722445 -0.131314 -0.460878].
Key (sphere_map_fn) not found.
Warning! No optimisation error threshold specified in config file (opt_max_err) - poor matches will not be dropped!
Key (sock_port) not found.
Key (com_port) not found.
Error! Recorder processing thread unable to set thread priority!
Key (thr_rgb_tfrm) not found.
Thresholding window size: 15 (ROI: 60 x 60)
Wrote 823 bytes to disk!
Error! Unable to set thread priority!
Starting sphere tracking loop!
Error! Unable to set thread priority!
Starting frame grabbing loop!
Error! Unable to set thread priority!
Frame captured 1920x1200x17301505 @ -1.000000 (1619520173168.530029)
Segmentation fault (core dumped)

I got to make some progress. This is the line I changed to get to this point: const GenApi::CFloatPtr camFrameRate = _cam.GetNodeMap().GetNode("ResultingFrameRate");

@lumyus
Copy link
Author

lumyus commented Apr 27, 2021

Fictrac now works with the Ace cameras! The last issue was the following:

 Pylon::CImageFormatConverter formatConverter;
        formatConverter.OutputPixelFormat = PixelType_BGR8packed;
        formatConverter.Convert(_pylonImg, _ptrGrabResult);

I will make a proper pull request once I cleaned everything up.
There is still a bug in the configGUI. I will keep you posted.

@DanTurner-Evans
Copy link

Now it's my turn to try to get fictrac up and running with an ace camera (on a Windows machine). I modified the cmake file to point to the most recent library (PylonBase_v6_3.lib) and everything compiled nicely. When I try to run the executable, however, I get the issues that @lumyus was having (a garbled camera name and NULL pointer dereferenced). @lumyus were you able to resolve everything and, if so, would you be willing to outline exactly what I should change? Thanks!

@DanTurner-Evans
Copy link

DanTurner-Evans commented May 31, 2022

As suggested above, I changed:
const GenApi::CFloatPtr camFrameRate = _cam.GetNodeMap().GetNode("ResultingFrameRate");
and added
formatConverter.OutputPixelFormat = PixelType_BGR8packed;
in BaslerSource.cpp, recompiled, and the camera seems to work now.

astraw added a commit to strawlab/fictrac that referenced this issue Jun 24, 2022
Closes rjdmoore#16 using the
suggestions there.
@DanTurner-Evans
Copy link

Had to make two additional changes:

  1. I had to update the library in CMakeLists to the most recent library
  2. I had to modify the congGui.cpp script to add the Basler option

@chiyu1203
Copy link

Hello, @DanTurner-Evans @lumyus
I followed this discussion and another one on Reddit to compile fictrac in Windows 11 with pylon 7.1.0
Reddit forum
Everything seemed to be working (fictrac is connected to Basler camera and track the ball as expected). However, I realised that the timestamp of the baseler camera is not saved in the .dat file.

Below is what I received from the fictrac file. If we pay attention to the row of BaslerSource::grab [DBG] Frame captured 300x270x17301505, we can see timestamp updated every frame but the other one seems to stuck at the value of the first timestamp

0.000968 ConfigParser::read [DBG] Extracted key: |vid_codec| val: |x264|
0.000985 ConfigParser::read [INF] Config file parsed (26 key/value pairs).
0.187349 BaslerSource::BaslerSource [INF] Opening Basler camera device: � �
0.206263 BaslerSource::BaslerSource [INF] Basler camera initialised (300x270 @ 144.009 fps)!
0.206310 ConfigParser::operator () [DBG] Key (output_fn) not found.
0.206315 ConfigParser::getStr [DBG] Key (fisheye) not found.
0.206373 Trackball::Trackball [DBG] Found sphere ROI centred at [0.039322 0.036111 0.998574], with radius 0.166490 rad.
0.206417 Trackball::Trackball [INF] Input sphere mask automatically generated using 1 ignore ROIs!
0.206422 Trackball::Trackball [DBG] roi_to_cam_r: 0.0361 -0.0393 0.0000
0.206429 Trackball::Trackball [DBG] Found C2A rotational transform: [-0.353763 -0.405708 1.550705].
0.206604 ConfigParser::getStr [DBG] Key (sphere_map_fn) not found.
0.206693 Trackball::Trackball [WRN] Warning! No optimisation error threshold specified in config file (opt_max_err) - poor matches will not be dropped!
0.207428 SocketRecorder::openRecord [INF] Opening UDP connection to 127.0.0.1:1111
0.207491 ConfigParser::operator () [DBG] Key (com_port) not found.
0.207505 Trackball::Trackball [WRN] Warning! Using default value for vid_codec (h264).
0.207759 Trackball::Trackball [DBG] Opening fictrac-raw-20231017_161559.avi for video writing (h264 300x270 @ 144.009217 FPS)
0.210025 ConfigParser::operator () [DBG] Key (thr_rgb_tfrm) not found.
0.210032 FrameGrabber::FrameGrabber [DBG] Thresholding window size: 15 (ROI: 60 x 60)
0.210154 FrameGrabber::process [DBG] Starting frame grabbing loop!
0.210162 FrameGrabber::process [DBG] Set frame grabbing thread priority to HIGH!
0.210293 ConfigParser::write [DBG] Wrote 926 bytes to disk!
0.210390 Trackball::process [DBG] Starting sphere tracking loop!
0.210398 Trackball::process [DBG] Set processing thread priority to HIGH!
0.210457 BaslerSource::grab [DBG] Frame captured 300x270x17301505 @ -1.000000 (472843433.523000)
0.211281 FrameGrabber::process [DBG] Processed frame added to input queue (l = 1).
0.211288 Trackball::process [INF] Frame 0
0.211310 Trackball::doSearch [INF] optimum sphere rotation: 0.000 0.000 0.000 (err=0.000e+00/its=0)
0.211313 Trackball::doSearch [DBG] Current sphere orientation: 0.000 0.000 0.000
0.211416 Trackball::updateSphere [DBG] Sphere ROI match overlap: 1.4%
0.211519 Trackball::process [INF] Timing grab/opt/map/plot/log/disp: 0.9 / 0.0 / 0.1 / 0.0 / 0.0 / 0.1 ms
0.211523 Trackball::process [INF] Average frame rate [in/out]: 0.0 [0.0 / 0.0] fps
0.216494 BaslerSource::grab [DBG] Frame captured 300x270x17301505 @ 472843433.523000 (472843439.559000)
0.217139 FrameGrabber::process [DBG] Processed frame added to input queue (l = 1).
0.217147 Trackball::process [INF] Frame 1
0.218896 Trackball::doSearch [INF] optimum sphere rotation: 0.000 0.000 0.000 (err=4.098e+03/its=20)
0.218900 Trackball::doSearch [DBG] Current sphere orientation: 0.000 0.000 0.000
0.218997 Trackball::updateSphere [DBG] Sphere ROI match overlap: 100.0%
0.219097 Trackball::process [INF] Timing grab/opt/map/plot/log/disp: 5.6 / 1.8 / 0.1 / 0.0 / 0.0 / 0.1 ms
0.219100 Trackball::process [INF] Average frame rate [in/out]: 33.0 [0.0 / 132.0] fps
0.224353 BaslerSource::grab [DBG] Frame captured 300x270x17301505 @ 472843433.523000 (472843447.417000)
0.225091 FrameGrabber::process [DBG] Processed frame added to input queue (l = 1).

In addition, I expected to see two types of timestamp saved in the .dat file. One at column 22 timestamp (since epoch) and the other one at column 25 alt. timestamp (since midnight) as described in the date_header.txt. However, in the .dat file, the first timestamp remained to be the first value of default timestamp(472843433.523000) and the second timestamp remained to be 0 throughtout the entire recording.
Have you seen error before? If so, what should I do to debug this?

@DanTurner-Evans
Copy link

Our .dat files are also missing the timestamp information. Luckily for our case, we send the data over the socket to a VR that runs in Unity and save the time information that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants