Flame drive blender smpl face model

face recognition to drive blender model to emotion and pose

This is a blender addon for driving the face model of smplx.It has two parts: face recognition(use FLAME) and face model driving(blender smplx addon).

Description

The blender addon part base on blender-smplx-addon download and face recognition part base on DECA

Getting Started

run blender addon frist, then local face recognition part

blender addon part

download the addon

• download the addon from here, or download from release page

blender load

• in blender load the addon

  

face recognition part

Clone the repo:

git clone https://github.com/YadiraF/DECA

Requirements

• face-alignment (Optional for detecting face, i use yolov8 pretrained)
  You can run

  pip install -r requirements.txt

• Prepare data

  download encoded pretrained model from here, put the file flame_encode_params.pkl under the data.

  yolov8 face pretrained model from here, put the file yolov8n_100e.pt under the data.

• run the face recognition part cd to the DECA folder

  python demo_video.py -i 0

quick start

• in blender make sure addon was loaded , and select View -> Sidebar, in Sidebar select Webcam

• in blender choose the smplx model to add

• push the webcam start botton

  

• run the face recognition part cd to the smpl_face_blender

  python demo_video.py -i 0

• more video can watch SMPL-X Blender Add-On -- Tutorial

• my addon video

TODO

• tracking on target face (multi-face situation tracking)
• more blender addon user interaction
• extend smplx model expression
• drive custom model
• merge two part(blender addon and face recognition) version,in blender install pytorch ?

Acknowledgements

For functions or scripts that are based on external sources, we acknowledge the origin individually in each file.
Here are some great resources we benefit:

• FLAME_PyTorch and TF_FLAME for the FLAME model
• blender
• smplx for blender model
• ultralytics for face dectection and recognition
• face_landmarks for face landmarks