main_evaluation.py run error

[zhangli3000]

Description:
main_evaluation.py run error about vio_performance_template.html

Command:

./evaluation/main_evaluation.py -r -a --plot --verbose_sparkvio --save_plots --save_boxplots --save_results ../main_ev.yaml

Console output:

Traceback (most recent call last):                                                                    
  File "./evaluation/main_evaluation.py", line 72, in <module>                                        
    if run(args):                                                                                     
  File "./evaluation/main_evaluation.py", line 21, in run                                             
    dataset_evaluator = DatasetEvaluator(experiment_params, args, extra_flagfile_path)                
  File "/usr/local/lib/python2.7/dist-packages/evaluation/evaluation_lib.py", line 291, in __init__   
    self.website_builder = evt.WebsiteBuilder(self.results_dir)                                       
  File "/usr/local/lib/python2.7/dist-packages/evaluation/tools/website_utils.py", line 27, in __init_
_                                                                                                     
    self.boxplot_template = self.env.get_template('vio_performance_template.html')                    
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 883, in get_template      
    return self._load_template(name, self.make_globals(globals))                                      
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 857, in _load_template    
    template = self.loader.load(self, name, globals)                                                  
  File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 115, in load                  
    source, filename, uptodate = self.get_source(environment, name)                                   
  File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 249, in get_source            
    raise TemplateNotFound(template)                                                                  
jinja2.exceptions.TemplateNotFound: vio_performance_template.html                                     

Additional files:
Please attach all the files needed to reproduce the error.

Please give also the following information:

• evo-1: https://github.com/ToniRV/evo-1
• Python version used: python2.7
• Ubuntu18.04
• did you change the source code? (yes / no):no

executable_path: '../Kimera-VIO/buil/stereoVIOEuroc'      
vocabulary_path: '../Kimera-VIO/vocabulary/ORBvoc.yml'    
results_dir: 'Kimera-VIO-Evaluation/reuslts'                                 
params_dir: '../Kimera-VIO/params/Euroc'                        
dataset_dir: '../Kimera-VIO/script/euroc'                  
                                                                                            
datasets_to_run:                                                                            
 - name: V1_01_easy                                                                         
   use_lcd: false                                                                           
   plot_vio_and_pgo: true                                                                   
   segments: [1]                                                                            
   pipelines: ['S']                                                                         
   discard_n_start_poses: 10                                                                
   discard_n_end_poses: 10                                                                  
   initial_frame: 10                                                                        
   final_frame: 22000                                                                       
   parallel_run: true                                                                       

[rginpan]

same error

[amihakko]

Description:
main_evaluation.py run error about vio_performance_template.html

Command:

./evaluation/main_evaluation.py -r -a --plot --verbose_sparkvio --save_plots --save_boxplots --save_results ../main_ev.yaml

Console output:

Traceback (most recent call last):                                                                    
  File "./evaluation/main_evaluation.py", line 72, in <module>                                        
    if run(args):                                                                                     
.....

I was able to resolve the issue. The fact is that in addition to the "website" package located in the:
Kimera-VIO-Evaluations/website
there is another website package located in the:
$Home/.local/lib/pythonX.X/site-packages/website , where X.X - your version Python.
If you copy the "templates" folder there, the problem will be solved.
Alternatively, you can tell Python to use the package that is in the Kimera-VIO-Evaluations/website folder first.

[geothan]

I have encountered the same problem and i tried to move templates folder to python directory, but the evaluation tool doesn't work. Have you solved the problem?

My yaml file is:

executable_path: '/home/george/catkin_ws/build/kimera_vio/stereoVIOEuroc'
vocabulary_path: '/home/george/catkin_ws/src/Kimera-VIO/vocabulary/ORBvoc.yml'
params_dir: '/home/george/catkin_ws/src/Kimera-VIO/params'
dataset_dir: '/home/george/datasets/EuRoC/ASL_Dataset_format'
results_dir: '/home/george/spark_vio_evaluation/results'
datasets_to_run:
name: V1_01_easy
use_lcd: true
plot_vio_and_pgo: true
segments: [1]
pipelines: ['S']
discard_n_start_poses: 0
discard_n_end_poses: 0
initial_frame: 10
final_frame: 220

the output of code execution before movement of folder templates:

(venv) george@pc:~/spark_vio_evaluation$ ./evaluation/main_evaluation.py -r -a --save_plots --save_results --save_boxplots experiments/george_example_euroc.yaml
Traceback (most recent call last):
File "./evaluation/main_evaluation.py", line 76, in
if run(args):
File "./evaluation/main_evaluation.py", line 25, in run
dataset_evaluator = DatasetEvaluator(experiment_params, args, extra_flagfile_path)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 298, in init
self.website_builder = evt.WebsiteBuilder(self.results_dir)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/tools/website_utils.py", line 29, in init
self.boxplot_template = self.env.get_template('vio_performance_template.html')
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/environment.py", line 883, in get_template
return self._load_template(name, self.make_globals(globals))
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/environment.py", line 857, in _load_template
template = self.loader.load(self, name, globals)
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/loaders.py", line 249, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: vio_performance_template.html

and the output of code execution after movement of folder templates:

(venv) george@pc:~/spark_vio_evaluation$ ./evaluation/main_evaluation.py -r -a --save_plots --save_results --save_boxplots experiments/george_example_euroc.yaml
0%| | 0/1 [00:00<?, ?it/s]I1116 22:53:56.135045 5048 evaluation_lib.py:305] Run dataset: V1_01_easy
Run pipeline: S
----------/home/george/catkin_ws/src/Kimera-VIO/params/S/flags/stereoVIOEuroc.flags: No such file or directory
E1116 22:53:56.236728 5048 evaluation_lib.py:168] Failed pipeline run!
Finished evaluation for dataset: V1_01_easy
I1116 22:53:56.237183 5048 evaluation_lib.py:308] Dataset: V1_01_easy failed!!
0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "./evaluation/main_evaluation.py", line 76, in
if run(args):
File "./evaluation/main_evaluation.py", line 26, in run
dataset_evaluator.evaluate()
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 309, in evaluate
raise Exception("Failed to run dataset %s." % dataset['name'])
Exception: Failed to run dataset V1_01_easy.

[amihakko]

I have encountered the same problem and i tried to move templates folder to python directory, but the evaluation tool doesn't work. Have you solved the problem?

My yaml file is:

executable_path: '/home/george/catkin_ws/build/kimera_vio/stereoVIOEuroc'
vocabulary_path: '/home/george/catkin_ws/src/Kimera-VIO/vocabulary/ORBvoc.yml'
params_dir: '/home/george/catkin_ws/src/Kimera-VIO/params'
dataset_dir: '/home/george/datasets/EuRoC/ASL_Dataset_format'
results_dir: '/home/george/spark_vio_evaluation/results'
datasets_to_run:
name: V1_01_easy
use_lcd: true
plot_vio_and_pgo: true
segments: [1]
pipelines: ['S']
discard_n_start_poses: 0
discard_n_end_poses: 0
initial_frame: 10
final_frame: 220

the output of code execution before movement of folder templates:

(venv) george@pc:~/spark_vio_evaluation$ ./evaluation/main_evaluation.py -r -a --save_plots --save_results --save_boxplots experiments/george_example_euroc.yaml
Traceback (most recent call last):
File "./evaluation/main_evaluation.py", line 76, in
if run(args):
File "./evaluation/main_evaluation.py", line 25, in run
dataset_evaluator = DatasetEvaluator(experiment_params, args, extra_flagfile_path)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 298, in init
self.website_builder = evt.WebsiteBuilder(self.results_dir)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/tools/website_utils.py", line 29, in init
self.boxplot_template = self.env.get_template('vio_performance_template.html')
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/environment.py", line 883, in get_template
return self._load_template(name, self.make_globals(globals))
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/environment.py", line 857, in _load_template
template = self.loader.load(self, name, globals)
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
File "/home/george/venv/local/lib/python2.7/site-packages/jinja2/loaders.py", line 249, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: vio_performance_template.html

and the output of code execution after movement of folder templates:

(venv) george@pc:~/spark_vio_evaluation$ ./evaluation/main_evaluation.py -r -a --save_plots --save_results --save_boxplots experiments/george_example_euroc.yaml
0%| | 0/1 [00:00<?, ?it/s]I1116 22:53:56.135045 5048 evaluation_lib.py:305] Run dataset: V1_01_easy
Run pipeline: S
----------/home/george/catkin_ws/src/Kimera-VIO/params/S/flags/stereoVIOEuroc.flags: No such file or directory
E1116 22:53:56.236728 5048 evaluation_lib.py:168] Failed pipeline run!
Finished evaluation for dataset: V1_01_easy
I1116 22:53:56.237183 5048 evaluation_lib.py:308] Dataset: V1_01_easy failed!!
0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "./evaluation/main_evaluation.py", line 76, in
if run(args):
File "./evaluation/main_evaluation.py", line 26, in run
dataset_evaluator.evaluate()
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 309, in evaluate
raise Exception("Failed to run dataset %s." % dataset['name'])
Exception: Failed to run dataset V1_01_easy.

You have dealt with the previous error, it seems thanks to my solution. And now you have a completely different error. It is connected with the fact that Kimera-Evaluate cannot detect parameter files. You need to rename the parameter "pipelines" in Yaml-file from ['S'] to ['Euroc'].

[geothan]

Thanks @amihakko , it seems to work. However it should have worked with parameters such as S, SP and SRP. Now, my execution stuck in that step (line 580) where there is an exception regarding traj_ref:

(venv) george@pc:~/spark_vio_evaluation$ ./evaluation/main_evaluation.py -r -a --save_plots --save_results --save_boxplots experiments/george_example_euroc.yaml
0%| | 0/1 [00:00<?, ?it/s]I1117 23:13:02.545836 7292 evaluation_lib.py:305] Run dataset: V1_02_medium
Run pipeline: Euroc
Successful pipeline run.
All pipeline runs were successful.
Finished evaluation for dataset: V1_02_medium
I1117 23:13:35.605365 7292 evaluation_lib.py:326] Evaluate dataset: V1_02_medium
Starting analysis of pipeline: Euroc
0%| | 0/1 [00:33<?, ?it/s]
Traceback (most recent call last):
File "./evaluation/main_evaluation.py", line 76, in
if run(args):
File "./evaluation/main_evaluation.py", line 26, in run
dataset_evaluator.evaluate()
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 313, in evaluate
self.evaluate_dataset(dataset)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 329, in evaluate_dataset
if not self.__evaluate_run(pipeline_type, dataset):
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 369, in __evaluate_run
dataset_name, discard_n_start_poses, discard_n_end_poses)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 407, in run_analysis
traj_ref, traj_est_vio, traj_est_pgo = self.read_traj_files(traj_ref_path, traj_vio_path, traj_pgo_path)
File "/home/george/venv/local/lib/python2.7/site-packages/evaluation/evaluation_lib.py", line 580, in read_traj_files
traj_ref = pandas_bridge.df_to_trajectory(pd.read_csv(traj_ref_path, sep=',', index_col=0))
AttributeError: 'module' object has no attribute 'df_to_trajectory'

Did you face that problem? What is the outcome of this execution? What does someone expect to see regarding VIO?

[amihakko]

@geothan , This is the problem of missing df_to_trajectory functions in the EVO library, namely the pandas_bridge.py file. To fix this, you need to remove your EVO library and install the EVO-1 fork, which is listed in the Prerequisites section, instead:
evo-1 // Fork from evo

[geothan]

@amihakko thanks a lot, i didn't noticed that! Now, the result files were generated and i moved on with jupyter notebook frontend. In Frontend Mono RANSAC, i get the below error, have you met this problem?

i followed up the below installation steps, including your suggested move of website folder:

sudo apt-get install virtualenv
virtualenv -p python2.7 ./venv
source ./venv/bin/activate

git clone https://github.com/ToniRV/Kimera-VIO-Evaluation
cd Kimera-VIO-Evaluation
pip install .
python setup.py develop

git clone https://github.com/ToniRV/evo-1.git
cd evo-1
pip install . --upgrade --no-binary evo
cd ..

Also, the path which i defined in Jupyter are:
vio_output_dir = "/home/george/Kimera-VIO-Evaluation/results/V1_01_easy/Euroc"
gt_data_file = "/home/george/datasets/EuRoC/ASL_Dataset_format/V1_01_easy/mav0/state_groundtruth_estimate0/data.csv"
left_cam_calibration_file = "/home/george/datasets/EuRoC/ASL_Dataset_format/V1_01_easy/mav0/cam0/sensor.yaml"

---

KeyError Traceback (most recent call last)
in ()
1 # Generate some trajectories for later plots
2 # Convert to evo trajectory objects
----> 3 traj_ref_unassociated = pandas_bridge.df_to_trajectory(gt_df)
4
5 # Use the mono ransac file as estimated trajectory.

/home/george/venv/local/lib/python2.7/site-packages/evo/tools/pandas_bridge.pyc in df_to_trajectory(df)
50 if not isinstance(df, pd.DataFrame):
51 raise TypeError("pandas.DataFrame or derived required")
---> 52 positions_xyz = df.loc[:,['x','y','z']].to_numpy()
53 quaternions_wxyz = df.loc[:,['qw','qx','qy','qz']].to_numpy()
54 # NOTE: df must have timestamps as index

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in getitem(self, key)
1492 except (KeyError, IndexError, AttributeError):
1493 pass
-> 1494 return self._getitem_tuple(key)
1495 else:
1496 # we by definition only have the 0th axis

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in _getitem_tuple(self, tup)
886 continue
887
--> 888 retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
889
890 return retval

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in _getitem_axis(self, key, axis)
1900 raise ValueError('Cannot index with multidimensional key')
1901
-> 1902 return self._getitem_iterable(key, axis=axis)
1903
1904 # nested tuple slicing

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in _getitem_iterable(self, key, axis)
1203 # A collection of keys
1204 keyarr, indexer = self._get_listlike_indexer(key, axis,
-> 1205 raise_missing=False)
1206 return self.obj._reindex_with_indexers({axis: [keyarr, indexer]},
1207 copy=True, allow_dups=True)

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in _get_listlike_indexer(self, key, axis, raise_missing)
1159 self._validate_read_indexer(keyarr, indexer,
1160 o._get_axis_number(axis),
-> 1161 raise_missing=raise_missing)
1162 return keyarr, indexer
1163

/home/george/venv/local/lib/python2.7/site-packages/pandas/core/indexing.pyc in _validate_read_indexer(self, key, indexer, axis, raise_missing)
1244 raise KeyError(
1245 u"None of [{key}] are in the [{axis}]".format(
-> 1246 key=key, axis=self.obj._get_axis_name(axis)))
1247
1248 # We (temporarily) allow for some missing keys with .loc, except in

KeyError: u"None of [Index([u'x', u'y', u'z'], dtype='object')] are in the [columns]"

[amihakko]

Glad I was able to help you with previous problems. But I haven't worked with Jupyter notebook, so I can't help here.