Facial-De-occlusion-Network

Quickstart in Colab

Model Architect

• This approach is an end to end inpainting method that learns to fill in user-specific details.
• No required 3D face to face reconstruction.
• Easy to train, customize, fine-tuning.

• Model Architecture
• Encoder and Decoder:
• Attention Module:

Datasets

Link Mask: Link

Link UnMask: Link

Structure of Reposity

• notebook: contains Jupyter notebook file to train, evaluate and inference the model
  • FaceDe_occlusion.ipynb: Step2step train model Face De-occlusion
• training_checkpoints: path to save checkpoints from model
• weights: model weight trained with no attention, can use to fine-tuning
• dataset: contains dataset splitted into train and test.
  • train: contain train set
    • target: target image
    • input: input image
  • test: contain test set
    • target: target image
    • input: input image

Results

• Train Based model (W/O Attention Module) with 100 epoch:

  • The generated image fills the lips and nose quite harmoniously. However, the image quality is blurry, not clear.

• Train with Attention Mudule with 300 epoch:

  • The generated image is clear and detailed, but there is a white part of the mouth that cannot be filled.

• Load Weights of Base model without Attention to Model with Attention and Fine-tune with 150 epoch:
  => Fixed problem with white part of the mouth.