questions about Scenario_runner and DReyeVR

[EvilQQQ]

Hello
I must admit that this is a very meaningful work. It provides great support for human factors safety engineering. We're trying to build our own scenes by using DReyeVR and scenario_runner. And we have some questions about using scenario_runner with DReyeVR. I read the issue96 and issue48. It may not help with us. **

1. How to reproduce the scenario like this. There are some scenarios that don't seem to be executing

We successfully run the example by using -route.xml and -json. After that, we choose the 13th route in routes_training.xml and all_towns_traffic_scenarios.json.

As shown in the figures, this route matches the scenario7. And the program run correctly to build the scenario7. But the simulator didn't trigger the scenario7. We used UE4 to locate the trigger point in the scenario7 and route13 in town03 and confirm that our VR vehicle passed through the trigger point, but it didn't have any effect.

2、As mentioned in the issues48, there are some scenarios which are not triggered correctly. Because my thesis needs to design some Customized VR scenarios about autonomous driving, i would like to design some scenarios in DReyeVR. If possible, would you mind give me some advice？

[ajdroid]

What version of scenario runner are you using? What is scenario 7 supposed to do?

[EvilQQQ]

Thanks for your reply.
1、Scenario runner version 0.9.13. 2、scenario7 for this.

We test the scenario1 3 4. They run successfully. As for scenario7, it can be created as scenario instance using scenario.py, and the actor spawned on the given trigger point. But it doesn't move.
2、I assumed that the scenario7 matches with carla AD challenge(https://carlachallenge.org/challenge/nhtsa/). So if we drive DRVRcarla to the trigger point. The scenario7 will be triggered and demonstrate like carlaAD Traffic Scenario 07.

[EvilQQQ]

here is the version scenario runner0.9.13 https://github.com/carla-simulator/scenario_runner/blob/v0.9.13/srunner/scenarios/route_scenario.py

[shh1v]

Have you tried running the scenario runner with the --debug flag? It tells you if it recognizes the trigger point or not.

[shh1v]

Also, can you post the scenario runner command with all the parameters you execute in the terminal so that I can reproduce the issue in my environment?

[EvilQQQ]

1、 Reply to the scenario runner command and Carla version @shh1v @ajdroid
python scenario_runner.py --route D:\CarlaDReyeVR\scenario_runner\srunner\data\routes_training.xml D:\CarlaDReyeVR\scenario_runner\srunner\data\all_towns_traffic_scenarios.json 0 --sync
I use the debug mode. And I'm sure that code works correctly. But it is not shown in the simulation environment. Intuitively, it doesn't perform well. If the ego vehicle is too fast, it doesn't trigger correctly. and another actor for this scenario will be stuck.

2、Questions for design some specific scenario for DRVRcarla.
I am pursuing to design specific scenario for DRVR to demonstrate better. Because the former scenario of scenario_runner 0.9.13 aimed at training and testing AI(only scenario1~10). like this

Additionally, it is not suitable for study human behaviors. We would like to kindly seek your assistance, as we have encountered a few challenges in our research and development process.

Firstly, please allow me to briefly describe the process of running a scenario.

(1) read the customed XXX.xml and XXX.json. like this
xml contains town_id, some waypoints for ego vehicle （according to UE4 coordinate） and route_id（Input as a parameter during execution)

json contains some trigger points of scenario.
like this

(2) generate route's trajectory according to waypoints from XXX.xml（interpolate and add the GPS route）

(3) Iterate through all scenario triggers (XXX.json) and waypoints to match by calculate the distance and yaw. then return a scenario set may be triggered.

(4) Build scenarios.
[1]during every scenario
initialize the environment. such as weather, friction and actors.
[2]create behavior tree by using Petree. (pyree is a package. I suppose it is described like multiprocessing or multithreading and it is built by atomic behavior designed by Carla in \ScenarioRunner\srunner\scenariomanager\scenarioatomics\automic_behaviors.py.
like this.

this is my customed behavior tree.
[3] Background Activity
[4]tick scenario() during a while circle.

We have successfully run our behavior tree on our own custom paths（xml） and triggers points（json）. But it was running too slowly. When I drive to the scenario trigger point, although the predefined frame is 50, CARLA experiences frame skipping alternating between fast and slow. Not only in our customed scenario, but also in Carla scenario_runner scenario 1～10. In particular, the human driving experience is crucial for studying human driving behavior. I wonder why the case is. Why games like GTA V, Pubg Need for Speed, Elden Ring, warcraft 3 or anything run with no lag 60fps as expected on a GPU like mine and even on weaker gpus, and a simple CAR VR simulator runs on fluctuations? Is this because of python solely? or Carla designing？ or VR, UE？Since this project was created with human behavior. I think the most essential thing is being able to render images with decent quality and frame rate to improve the experience of human driving.

Some hypotheses.
1、 shall we optimize the behavior tree？
2、 purchase more powerful hardware？
We now use the RTX4060 8G and i9-14900k. subsequently, we will use 4090 24G. If cost is not a concern, switching to a workstation with multiple CPUs and more powerful graphics cards can indeed be beneficial for running complex scenes？

[shh1v]

Thank you for the detailed explanation. I have been working for the past several months integrating scenario runner in our pipeline for conducting behavioral research. We have implemented SAE level 3 vehicle behavior, where the driver is required to intervene and take physical control upon the triggering of scenarios executed by scenario runner. I am happy to help you out but you may want to review previous discussions (like this) on CARLA and SR GitHub repositories as this issue is very well documented and solutions to make it working have been proposed. Mainly its an issue with how and why CARLA is developed. One thing I would also like to point out that there is a good chance, generally speaking, that there is an issue with your behavior tree (particularly py_trees ParallelPolicy) as we too had many issues with getting the scenarios working as a result of not understanding how it works. I am glad to reproduce the simulation if you could provide your code.

[EvilQQQ]

Thanks!!! and I apologize for the delayed response due to some recent paper submission matters.
1、I have understood the issues with the hardware. Are you referring to the vehicle behavior that has achieved SAE level 3, as implemented in the CarlaDRVR project? That sounds cool, and it's exactly what we need for our work. We have always wanted to achieve the scenarios (human-machine shared) you mentioned. How did you manage to do it? Could you share your experience?

2、I have already sent you the running video(In order to stabilize the frame rate, eye tracking has not been activated for the time being.), customized XML route, JSON scene trigger points, and behavior tree that I modified, in the form of an email. You can directly replace the cut_in.py file in \CarlaDReyeVR\scenario_runner\srunner\scenarios with my py file. Then, import it in \CarlaDReyeVR\scenario_runner\srunner\scenarios\route_scenario.py and make appropriate modifications, like this.

you can run it by this command.
python --route D:\CarlaDReyeVR\scenario_runner\srunner\data\custom_routes.xml D:\CarlaDReyeVR\scenario_runner\srunner\data\custom_scenarios.json 100 --sync

additionally, our frame is 60. and we set the avoid_collision = True (in \srunner\scenaromanager\scenarioatomics\atomic_behaviors.py\Class WaypointFollower)

Thank you again for being willing to help us check the behavior tree to see if it is the reason for the varying running speed in our scenarios. (Although the displayed frame rate remains constant at 60, the running speed fluctuates between fast and slow which is very noticeable to the driver.)

)

[EvilQQQ]

@shh1v and if have any problems about my code, contact me anytime.

[EvilQQQ]

@shh1v code and video are here. https://github.com/EvilQQQ/VR_scenario