Skip to content

Latest commit

 

History

History
86 lines (65 loc) · 2.69 KB

README.md

File metadata and controls

86 lines (65 loc) · 2.69 KB
Raw

KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

This repo covers the data generation, training and inference of kovis visual servos introduced in KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation (IROS 2020) by En Yen Puang, Keng Peng Tee and Wei Jing.

Inference uses UR5, realsense D435 camera, ROS and urx.

[ Paper ] [ Video ]

demo

Installation

We recommend using virtual python environment like Conda with python3.

  1. Install PyTorch
  2. Install other packages
pip install -r requirements.txt
  1. (Optinal) Install ROS, rospkg and RealSense camera driver
  2. (Optinal) install urx for UR robot
pip install git+https://github.com/enyen/python-urx

Usage

  1. Generate training data in pyBullet:
cd KOVIS_VisualServo
# example for generating dataset for pick-mug task
python oja_pick.py cfg/dataset_pick_mug.yaml

If no monitor is connected, render without shadow by replacing line 19 with

pyb.connect(pyb.DIRECT)  # pyb.connect(pyb.GUI)
  1. Training servo in pyTorch:
# example for training for pick-mug task
python train_servo.py cfg/train_pick_mug.yaml
  1. Running on robot:
    • launch realsense camera with both infra cameras enabled
    • turn off realsense laser using rqt_reconfigure
from inference_oja import Interface
rob = Interface()

# reach
# TODO: move tcp to close to object

# pick
rob.servo('pick_mug', 10, [0.01, 0.01, 0.01, 0.05, 0, 0], [0.1, 5])
rob.set_gripper(1)

# continue
# TODO: move object away

Citation

Please cite our paper if you use this code.

  @inproceedings{puang2020kovis,
    title={KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation},
    author={Puang, En Yen and Tee, Keng Peng and Jing, Wei},
    booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    pages={7527--7533},
    year={2020},
    organization={IEEE}
  }

License

This code is under GPL-3.0 License. Contact

puangenyen at gmail . com

to discuss other license agreement.