D2S: Representing sparse descriptors and 3D coordinates for camera relocalization

Project Page | Paper

D2S: Representing sparse descriptors and 3D coordinates for camera relocalization
Bach-Thuan Bui, Huy-Hoang Bui, Dinh-Tuan Tran, Joo-Ho Lee
IEEE Robotics and Automation Letters (RAL)

• Introduction
• Installation
• Dataset
  • Supported Datasets
  • Download preprocessed datasets
• Evaluation
• Training
• Run with your own dataset
• BibTex Citation

Introduction

D2S is a learning-based visual re-localization method. It concentrates on learning to generate 3D scene coordinates from sparse descriptors extracted from a single image. Once trained for a specific sparse SfM scene, D2S can accurately estimate the camera’s position and orientation from a new image of the scene. D2S also serves as a natural filter for outliers and non-robust descriptors, resulting in an enhanced localization accuracy down the line.

This repository contains the Pytorch implementation of our papers:

• D2S: Representing Sparse Descriptors and 3D Coordinates for Camera Relocalization
• Fast and Lightweight Scene Regressor for Camera Relocalization (feat2map branch)

Todo list:

• release code
• run with custom data
• add inference with pre-trained models

Installation

D2S is based on PyTorch. The main framework is implemented in Python. Python 3.9 + required packages

git clone https://github.com/ais-lab/d2s.git
cd d2s
conda create --name d2s python=3.9
conda activate d2s
# Refer to https://pytorch.org/get-started/previous-versions/ to install pytorch compatible with your CUDA
python -m pip install torch==1.12.0 torchvision==0.13.0 
python -m pip install -r requirements.txt

D2S uses hierarchical localization toolbox(hloc) to label descriptors coordinates. However, we provided scripts to download all preprocessed datasets for an immediate run.

Dataset

Supported Datasets

• 7scenes
• Indoor6
• Cambridge Landmarks
• BKC Ritsumeikan

Download preprocessed datasets

Please run the provided scripts to prepare and download the data which has been preprocessed by running:

7scenes

./prepare_scripts/seven_scenes.sh

Cambridge Landmarks

./prepare_scripts/cambridge.sh 

Indoor-6

./prepare_scripts/indoor6.sh

Evaluation with pre-trained models

Please download the pre-trained models by running:

./prepare_scripts/download_pre_trained_models.sh

For example, to evaluate Indoor6 scene1:

python runners/eval.py --dataset indoor6 --scene scene1 -expv d2s

You can receive a result like this:

For pre-trained models of 7scenes, you can download it from here

Training

python runners/train.py --dataset indoor6 --scene scene1 -expv d2s

Run with your own dataset

Install COLMAP and Hloc

• Install COLMAP from source or just run:

sudo snap install colmap

• Install Hloc

git clone --recursive https://github.com/thuanaislab/Hierarchical-Localization/
cd Hierarchical-Localization/
conda activate d2s
python -m pip install -e .
cd ..

Prepare your own data

Create a custom_dataset folder as follow and put all images in images folder:

d2s
├── custom_dataset
│   ├── images
|   |   ├── image_1.jpg
|   |   ├── ...
|   |   └── image_N.jpg
├── ...
└── README.md

Run entire pipeline on your own data

Then run this command:

./run_custom_data_pipeline.sh custom_dataset/

BibTex Citation

If you find this project useful, please consider citing:

@article{bui2024d2s,
  title={D2S: Representing sparse descriptors and 3D coordinates for camera relocalization},
  author={Bui, Bach-Thuan and Bui, Huy-Hoang and Tran, Dinh-Tuan and Lee, Joo-Ho},
  journal={IEEE Robotics and Automation Letters},
  year={2024}
}
@article{bui2022fast,
  title={Fast and Lightweight Scene Regressor for Camera Relocalization},
  author={Bui, Thuan B and Tran, Dinh-Tuan and Lee, Joo-Ho},
  journal={arXiv preprint arXiv:2212.01830},
  year={2022}
}