| Title: |
Facial Landmark Feature Fusion in Transfer Learning of Child Facial Expressions |
| Authors: |
Witherow, Megan A.; Samad, Manar D.; Diawara, Norou; Iftekharuddin, Khan M. |
| Source: |
Electrical & Computer Engineering Faculty Publications |
| Publisher Information: |
ODU Digital Commons |
| Publication Year: |
2022 |
| Collection: |
Old Dominion University: ODU Digital Commons |
| Subject Terms: |
Data modeling; Image classification; Feature extraction; Convolutional neural networks; Information fusion; Machine learning; Facial expression recognition; Transfer learning; Feature fusion; Facial landmarks; Child facial expressions; Artificial Intelligence and Robotics; Data Science; Electrical and Computer Engineering |
| Description: |
Automatic classification of child facial expressions is challenging due to the scarcity of image samples with annotations. Transfer learning of deep convolutional neural networks (CNNs), pretrained on adult facial expressions, can be effectively finetuned for child facial expression classification using limited facial images of children. Recent work inspired by facial age estimation and age-invariant face recognition proposes a fusion of facial landmark features with deep representation learning to augment facial expression classification performance. We hypothesize that deep transfer learning of child facial expressions may also benefit from fusing facial landmark features. Our proposed model architecture integrates two input branches: a CNN branch for image feature extraction and a fully connected branch for processing landmark-based features. The model-derived features of these two branches are concatenated into a latent feature vector for downstream expression classification. The architecture is trained on an adult facial expression classification task. Then, the trained model is finetuned to perform child facial expression classification. The combined feature fusion and transfer learning approach is compared against multiple models: training on adult expressions only (adult baseline), child expression only (child baseline), and transfer learning from adult to child data. We also evaluate the classification performance of feature fusion without transfer learning on model performance. Training on child data, we find that feature fusion improves the 10-fold cross validation mean accuracy from 80.32% to 83.72% with similar variance. Proposed fine-tuning with landmark feature fusion of child expressions yields the best mean accuracy of 85.14%, a more than 30% improvement over the adult baseline and nearly 5% improvement over the child baseline. |
| Document Type: |
conference object |
| File Description: |
application/pdf |
| Language: |
unknown |
| Relation: |
https://digitalcommons.odu.edu/ece_fac_pubs/353; https://digitalcommons.odu.edu/context/ece_fac_pubs/article/1362/viewcontent/Iftekharuddin_2022_FacialLandmarkFeatureFursioninTransferOCR.pdf |
| DOI: |
10.1117/12.2641898 |
| Availability: |
https://digitalcommons.odu.edu/ece_fac_pubs/353; https://doi.org/10.1117/12.2641898; https://digitalcommons.odu.edu/context/ece_fac_pubs/article/1362/viewcontent/Iftekharuddin_2022_FacialLandmarkFeatureFursioninTransferOCR.pdf |
| Rights: |
Copyright © 2022 Society of Photo‑Optical Instrumentation Engineers (SPIE). One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this publication for a fee or for commercial purposes, and modification of the contents of the publication are prohibited. Included in accordance with publisher policy. |
| Accession Number: |
edsbas.7C973482 |
| Database: |
BASE |