Project Page | arXiv:2409.11635
Due to the privacy policies of the BioVid Database, we can only release the checkpoints, training code, and inference code. To minimize our effort, we are releasing the training and preprocessing code as is for reference purposes. We have only tested and verified the inference code, which includes a Gradio-based demo.
Install Inferno for the EMOCA decoder. Follow the instructions here and download the necessary models for facial reconstruction here. We have slightly modified the original code to generate useful latent variables for the face reconstruction app and to support the render_from_exp.py script.
Note:
Installing pytorch3d might present compatibility issues due to mismatched versions of CUDA, PyTorch, and pytorch3d. If this occurs, install pytorch3d separately.
# Create and activate a new Conda environment
conda create python=3.10 -n paindiff
conda activate paindiff
# Install the required packages and pytorch3d
# Ensure the CUDA version matches your PyTorch installation and system configuration: https://pytorch.org/get-started/locally/
pip install -r requirements.txt
FORCE_CUDA=1 pip install git+https://github.com/facebookresearch/pytorch3d.git@stable
# Clone and set up the Inferno package
git clone https://github.com/damtien444/inferno inferno_package
cd inferno_package/
bash pull_submodules.sh
pip install -e .
# Download the pretrained EMOCA model for PainDiffusion
cd inferno_apps/FaceReconstruction
bash download_assets.shScreencast.from.11-21-2024.03.40.59.PM.webm
-
Download the Model Checkpoint
Download Link -
Place the Checkpoint
Save the downloaded checkpoint in a directory of your choice. -
Update Configuration
Editconfigure/sample_config.ymland set theBEST_CKPTfield to the path of your checkpoint. -
Run the Demo
python online_run.py
-
Access the Demo
Open http://127.0.0.1:7860/ in your web browser.
Installing pytorch3d can be tricky, especially if your system has multiple versions of CUDA. Use the following commands to set the appropriate CUDA version for your PyTorch installation:
# Replace {cuda_version} with the correct version for your system
export CUDA_HOME=/usr/local/cuda-{cuda_version}
export CUDA_PATH=$CUDA_HOME
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATHThis project is heavily based on the excellent implementations of diffusion models from: