MULTI-FEATURE AWARE POSE AND GEOMETRY BASED FACIAL EXPRESSION RECOGNITION USING DEEP LEARNING

Ankit Sharma Department of Computer Science and Engineering, Delhi Technological University, ankitsharma_2k20cse05@dtu.ac.in
Dr. Manoj Sethi Department of Computer Science and Engineering, Delhi Technological University

Abstract

Facial expressions are the primary way to express intentions and emotions. In real life, computers can understand the emotions of humans by analysing their facial expressions. Facial Expression Recognition (FER) plays an important role in human-computer interaction necessity and medical field [1]. In the past, the facial features are extracted manually for recognizing the expressions. In the present, it is an important hotspot in computer vision, Internet of Things, and artificial intelligence fields. Certain processes are involved to recognize the facial expression in an efficient manner such as, • Preprocessing • Segmentation • Feature Extraction • Classification Various algorithms are designed manually for feature extraction and other feature extraction algorithms such as Local Binary Pattern (LBP), Gabor wavelet, Histogram of Oriented Gradient (HOG) [2],[3]. Various challenges are involved in the FER to recognize accurate expressions of facial images. For robust classification of facial expression, consideration of illumination and pose of the facial image is important. The poses and facial identity learning are essential to get accurate results. Several existing works faced challenges regarding identity, pose variation, and inter-subject variation. For estimating the pose of the facial images existing methods used hand-crafted features. For detecting the pose of the facial images, pose normalization is performed by considering the angle in the existing works [3]. Previous works also considering pixel-based normalization to increase the accuracy for recognizing the facial expression [2]. On the other hand, general illumination effect of the images also affects the accuracy, by contrast, occlusion, etc. Segmentation is one of the major processes to partition the facial images to extract the features. Various existing methods used different techniques to segment the facial images such as bounding box-based segmentation, region-based segmentation, cluster-based segmentation, etc., by using various algorithms such as Discrete Cosine Transform (DCT), K-means Clustering algorithm, etc. For extracting the features after segmenting the facial images various Machine learning (ML) algorithms such as Support Vector Machines (SVM), Naive Bayes, K-Nearest Neighbours (KNN), and Deep Learning (DL) algorithms such as Convolutional Neural network (CNN), Generative Adversarial Network (GAN), Long-Short Term Memory Networks (LSTM) [2], [3], [4]. After extracting the features from the facial images, classification is performed to identify the facial expressions (happy, sad, anger etc.). Various classifiers are used in the previous works for classification processes such as VGG-16, VGG-19, ResNet, etc [2], [4]. These classifiers get the extracted feature as input and classify the expressions. For training and testing purposes various datasets are used but, mostly FER-2013, CK+, and JAFFE datasets are used [1], [2], [3].

Keywords:

: Facial Expression Recognition, Deep Learning, Capsule Neural Network, Fuzzy C-means Clustering, Segmentation, Facial Landmark Detection


Full Text:

PDF


References


[1] Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021). Facial Expression Recognition Using Hybrid Features of Pixel and Geometry. IEEE Access, 9, 18876-18889. [2] Mohan, K., Seal, A., Krejcar, O., & Yazidi, A. (2021). Facial Expression Recognition Using Local Gravitational Force Descriptor-Based Deep Convolution Neural Networks. IEEE Transactions on Instrumentation and Measurement, 70, 1-12. [3] Liu, J., Feng, Y., & Wang, H. (2021). Facial Expression Recognition Using Pose-Guided Face Alignment and Discriminative Features Based on Deep Learning. IEEE Access, 9, 69267-69277. [4] Tsai, K., Tsai, Y., Lee, Y., Ding, J., & Chang, R.Y. (2021). Frontalization and adaptive exponential ensemble rule for deep-learning-based facial expression recognition system. Signal Process. Image Commun., 96, 116321. [5] Kim, H., Kim, H., Rew, J., & Hwang, E. (2020). FLSNet: Robust Facial Landmark Semantic Segmentation. IEEE Access, 8, 116163-116175. [6] Chen, A., Xing, H., & Wang, F. (2020). A Facial Expression Recognition Method Using Deep Convolutional Neural Networks Based on Edge Computing. IEEE Access, 8, 49741-49751. [7] Liu, Y., Dai, W., Fang, F., Chen, Y., Huang, R., Wang, R., & Wan, B. (2021). Dynamic multi-channel metric network for joint pose-aware and identity-invariant facial expression recognition. Information Sciences, 578, 195-213. [8] Sepas-Moghaddam, A., Etemad, A., Pereira, F., & Correia, P.L. (2021). CapsField: Light Field-Based Face and Expression Recognition in the Wild Using Capsule Routing. IEEE Transactions on Image Processing, 30, 2627-2642. [9] Tang, Y., Zhang, X., Hu, X., Wang, S., & Wang, H. (2021). Facial Expression Recognition Using Frequency Neural Network. IEEE Transactions on Image Processing, 30, 444-457. [10] Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2020). Geometry Guided Pose-Invariant Facial Expression Recognition. IEEE Transactions on Image Processing, 29, 4445-4460. [11] Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2020). A Unified Deep Model for Joint Facial Expression Recognition, Face Synthesis, and Face Alignment. IEEE Transactions on Image Processing, 29, 6574-6589. [12] Zhang, J., Hu, H., & Feng, S. (2020). Robust Facial Landmark Detection via Heatmap-Offset Regression. IEEE Transactions on Image Processing, 29, 5050-5064.

Refbacks

  • There are currently no refbacks.