Skip to content

Latest commit

 

History

History
107 lines (99 loc) · 3.79 KB

README.md

File metadata and controls

107 lines (99 loc) · 3.79 KB
Raw

Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video (CVPR 2023)

Project Page | Paper | Video

Prerequisite

  • Create a new enviornment by 
    conda env create -f env.yml
    or (recommend) 
    conda create -n handavatar python=3.9
conda activate handavatar
pip install -r requirements.txt
  • Install leap from its official repository

Download

  • Download offical InterHand2.6M.

  • You should accept MANO LICENCE and download MANO model from official website.

  • Link floder

    ln -s /path/to/mano_v1_2 mano
cd mano
ln -s models/MANO_RIGHT.pkl MANO_RIGHT.pkl
cd .. && mkdir data && cd data
ln -s /path/to/InterHand2.6M InterHand

  • Hand segmentation (Unnecessary, If only runing inference): MANO meshes are projected to generate hand masks.

    python segment/seg_interhand2.6m_from_mano.py

    Set subject in the file for different subjects.

  • Download pretrained model and pre-processed data from Google Drive.

    The data can also be pre-processed by python handavatar/core/data/interhand/train.py after generating hand segmentation mask.

  • The overall floder is as follows

    ROOT
    ├──data
        ├──InterHand
            ├──5
                ├──InterHand2.6M_5fps_batch1
                    ├──images
                    ├──preprocess
                    ├──masks_removeblack
                |──annotations
    ├──handavatar
        ├──out/handavatar/interhand
            ├──test_Capture0_ROM04_RT_Occlusion
                ├──pretrained_model/latest.tar
            ├──test_Capture1_ROM04_RT_Occlusion
                ├──pretrained_model/latest.tar
            ├──val_Capture0_ROM04_RT_Occlusion
                ├──pretrained_model/latest.tar
    ├──smplx
        ├──out/pretrained_lbs_weights
            ├──lbs_weights.pth
    ├──pairof
        ├──out/pretrained_pairof
            ├──pairof.ckpt
    ├──mano

Inference

./handavatar/scripts/run_hand.sh
  • Results are saved in the same floder of pretrained model.

Metrics

Set the following configs in the config file

phase: 'val'
experiment: 'pretrained_model'
resume: True

then run

./handavatar/scripts/train_hand.sh

Training

Set the following configs in the config file

phase: 'train'
experiment: 'your_exp_name'
resume: False

then run

./handavatar/scripts/train_hand.sh
  • Tensorbard can be to track training process.
  • The training need one GPU and >32G memory. You can reduce patch: N_patches or patch: size to reduce GPU usage . In the mean time, incearsing train: accum_iter can control batch size (i.e., batch_size=N_patches*accum_iter) in an optimization step.

Reference

@inproceedings{bib:handavatar,
  title={Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video},
  author={Chen, Xingyu and Wang, Baoyuan and Shum, Heung-Yeung},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}

Acknowledgement

Our implementation is based on COAP, SMLPX, and HumanNeRF. We thank them for inspiring implementations.