Review of 3D Facial Expression Acquisition and Modeling Technology

doi:10.16182/j.issn1004731x.joss.201807001

Abstract

Abstract: Face is an important biological feature of human. High-fidelity 3D face construction is widely applied and has attracted the attention of researchers for a long time. Now the mature 3D model acquisition still depends on the expensive hardware and controlled lighting environment in the lab. However, academic focus has gradually transformed to researches based on daily living environment and low cost equipment. According to the source of input data, the existing works are divided into two categories: works under controlled environment and uncontrolled environment. Methods are summarized and classified based on the categories. Existing problems and future research directions are discussed in the end.

Key words: 3D face acquisition, 3D face modeling, controlled environment, uncontrolled environment, coarse face model, fine-detailed face model, feature detection

CLC Number:

TP391.9

Wang Shan, Shen Xukun, Zhao Qinping. Review of 3D Facial Expression Acquisition and Modeling Technology[J]. Journal of System Simulation, 2018, 30(7): 2423-2444.

References

[1] J Roth, Y Tong, Liu X. Adaptive 3D Face Reconstruction from Unconstrained Photo Collections[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (S0162-8828), 2017, 39(11): 2127-2141.
[2] Vicon. Vicon Cara System[EB/OL].[2018-05-16]. https://www.vicon.com/products/camera-systems/cara-1.
[3] OpTiTrack. OptiTrack Expression[EB/OL]. [2018-05-16]. http://optitrack.com/products/expression/indepth.html.
[4] Guenter B, Grimm C, Wood D, et al.Making faces[C]// Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 1998, ACM. 1998: 55-66.
[5] Bickel B, Botsch M, Angst R, et al.Multi-scale capture of facial geometry and motion[J]. ACM Trans. Graph.(S0730-0301), 2007, 26(3): 33.
[6] Huang H, Chai J, Tong X, et al.Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition[J]. ACM Trans. Graph.(S0730-0301), 2011, 30(4): 1-10.
[7] Beeler T, Bickel B, Beardsley P, et al.High-quality single-shot capture of facial geometry[J]. ACM Trans. Graph.(S0730-0301), 2010, 29(4): 1-9.
[8] Beeler T, Hahn F, Bradley D, et al.High-quality passive facial performance capture using anchor frames[C]// ACM SIGGRAPH 2011 papers. 2011, ACM: Vancouver, British Columbia, Canada. 2011: 1-10.
[9] Bradley D, Heidrich W, Popa T, et al.High resolution passive facial performance capture[J]. ACM Trans. Graph.(S0730-0301), 2010, 29(4): 1-10.
[10] Zhang L, Snavely N, Curless B, et al.Spacetime faces: high resolution capture for modeling and animation[J]. ACM Trans. Graph.(S0730-0301), 2004, 23(3): 548-558.
[11] Wang Y, Huang X, Lee C, et al.High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions[J]. Computer Graphics Forum (S0167-7055), 2004, 23(3): 677-686.
[12] Li H, Adams B, Guibas L, et al.Robust single-view geometry and motion reconstruction[J]. ACM Trans. Graph.(S0730-0301), 2009, 28(5): 1-10.
[13] Ma W C, Hawkins T, Peers P, et al.Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination[C]//Eurographics Conference on Rendering Techniques, 2007: 183-194.
[14] Debevec P, Hawkins T, Tchou C, et al.Acquiring the reflectance field of a human face[C]//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000, ACM Press/Addison- Wesley Publishing Co. 2000: 145-156.
[15] Hawkins T, Wenger A, Tchou C, et al.Animatable Facial Reflectance Fields[C]//Rendering Techniques ’04 (Proceedings of the Second Eurographics Symposium on Rendering, 2004: 309-321.
[16] Ma W C, Jones A, Chiang J Y, et al.Facial performance synthesis using deformation-driven polynomial displacement maps[J]. ACM Trans. Graph.(S0730-0301), 2008, 27(5): 1-10.
[17] Ghosh A, Fyffe G, Tunwattanapong B, et al.Multiview face capture using polarized spherical gradient illumination[J]. ACM Trans. Graph.(S0730-0301), 2011, 30(6): 1-10.
[18] Blanz V, T Vetter. A morphable model for the synthesis of 3D faces[C]//Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 1999, ACM Press/Addison-Wesley Publishing Co. 1999: 187-194.
[19] Cao C, Weng Y, Zhou S, et al.FaceWarehouse: A 3D Facial Expression Database for Visual Computing[J]. IEEE Transactions on Visualization & Computer Graphics (S1077-2626), 2014, 20(3): 413-425.
[20] Cootes T F, Taylor C J.Statistical Models of Appearance for computer vision[C]//Proceedings of SPIE - The International Society for Optical Engineering, 2000. 4322(1): 236-248.
[21] Matthews I, Baker S.Active appearance models revisited[J]. International Journal of Computer Vision (S0920-5691) , 2004, 60(2): 135-164.
[22] Cristinacce D, Cootes T.Automatic feature localisation with constrained local models[J]. Pattern Recognition (S0031-3203), 2008, 41(10): 3054-3067.
[23] Cristinacce D, Cootes T F.Feature Detection and Tracking with Constrained Local Models[C]//British Machine Vision Conference 2006, Edinburgh, UK, 2006.
[24] Tulyakov S, Sebe N.Regressing a 3D Face Shape from a Single Image[C]//2015 IEEE International Conference on Computer Vision (ICCV), Santiago Chile: IEEE, 2015: 3748-3755.
[25] Tzimiropoulos G.Project-Out Cascaded Regression with an application to face alignment[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015: 3659-3667.
[26] Sun Y, Wang X, Tang X.Deep Convolutional Network Cascade for Facial Point Detection[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 2013: 3476-3483.
[27] Wu Y, Wang Z, Ji Q.Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 2013: 3452-3459.
[28] Luo P, Wang X, Tang X.Hierarchical face parsing via deep learning[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA: IEEE, 2012: 2480-2487.
[29] Zhang J, Shan S, Kan M, et al.Coarse-to-Fine Auto-Encoder Networks (CFAN) for Real-Time Face Alignment[C]//Computer Vision-ECCV 2014. Cham: Springer International Publishing. 2014: 1-16.
[30] Hao W L, Xiaoming, Doretto G.Face alignment via boosted ranking model[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK, USA: IEEE, 2008: 1-8.
[31] Pandzic I S, Forchheimer R.MPEG-4 Facial Animation: The Standard, Implementation And Applications[M]. John Wiley & Sons, Inc., 2002: 299.
[32] Ekman P, Friesen W V.Facial Action Coding System (FACS): a Technique for the Measurement of Facial Actions[J]. Rivista Di Psichiatria (S2038-2502), 1978, 47(2): 126-38.
[33] Chuang E S, Deshpande H, Bregler C.Facial Expression Space Learning[C]//Pacific Conference on Computer Graphics and Applications. Beijing, China: IEEE, 2002: 68.
[34] Pighin F, Hecker J, Lischinski D, et al.Synthesizing realistic facial expressions from photographs[C]// ACM SIGGRAPH 2005 Courses. 2005, ACM: Los Angeles, California. 2005.
[35] Pyun H, Kim Y, Chae W, et al.An example-based approach for facial expression cloning[C]//ACM SIGGRAPH 2006 Courses. 2006, ACM: Boston, Massachusetts, 2006.
[36] Chu B, S Romdhani, Chen L. 3D-Aided Face Recognition Robust to Expression and Pose Variations[C]//IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA: IEEE, 2014: 1907-1914.
[37] L Yin, X Chen, Y Sun, et al.A high-resolution 3D dynamic facial expression database[C]//2008 8th IEEE International Conference on Automatic Face Gesture Recognition. Amsterdam, Netherlands: IEEE, 2008: 1-6.
[38] 3dMD. 3D Surface Imaging System [EB/OL]. [2018-05-16]. http://www.3dmd.com/.
[39] Vlasic D, Brand M, Pfister H, et al.Face transfer with multilinear models[C]//ACM SIGGRAPH 2006 Courses. 2006, ACM: Boston, Massachusetts. 2006.
[40] Kemelmacher-Shlizerman I, S M Seitz. Face reconstruction in the wild[J]. IEEE International Conference on Computer Vision, 2011, 58(11): 1746-1753.
[41] J Roth, Y Tong, Liu X. Unconstrained 3D face reconstruction[C]//Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE, 2015: 2606-2615.
[42] Ruo Z, Ping-Sing T, J.E.C, et al. Shape-from-shading: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (S0162-8828), 1999, 21(8): 690-706.
[43] Decarlo D, Metaxas D.The integration of optical flow and deformable models with applications to human face shape and motion estimation[C]//Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco, CA, USA: IEEE, 1996.
[44] Brand M, Bhotika R.Flexible flow for 3D nonrigid racking and shape recovery[C]//Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. Kauai, HI, USA: IEEE, 2001.
[45] Borshukov G, Piponi D, Larsen O, et al.Universal capture: image-based facial animation for "The Matrix Reloaded"[C]//ACM SIGGRAPH 2006 Courses. 2006, ACM: Boston, Massachusetts, 2006.
[46] Valgaerts L, Wu C, Bruhn A, et al.Lightweight binocular facial performance capture under uncontrolled lighting[J]. ACM Trans. Graph.(S0730-0301), 2012, 31(6): 1-11.
[47] Weise T, Bouaziz S, Li H, et al.Realtime performance-based facial animation[J]. ACM Trans. Graph.(S0730-0301), 2011, 30(4): 1-10.
[48] Fyffe G, Jones A, Alexander O, et al.Driving High-Resolution Facial Scans with Video Performance Capture[J]. ACM Trans. Graph.(S0730-0301), 2014, 34(1): 1-14.
[49] Garrido P, Valgaert L, Wu C, et al.Reconstructing detailed dynamic face geometry from monocular video[J]. ACM Trans. Graph.(S0730-0301), 2013, 32(6): 1-10.
[50] Li H, Yu J, Ye Y, et al., Realtime facial animation with on-the-fly correctives[J]. ACM Trans. Graph.(S0730-0301), 2013, 32(4): 1-10.
[51] Bouaziz S, Wang Y, Pauly M.Online modeling for realtime facial animation[J]. ACM Trans. Graph.(S0730-0301), 2013, 32(4): 1-10.
[52] Newcombe R A, Izadi S, Hilliges O, et al.KinectFusion: Real-time dense surface mapping and tracking[C]//2011 10th IEEE International Symposium on Mixed and Augmented Reality. Basel, Switzerland: IEEE, 2011.
[53] Da F, Sui Y.3D reconstruction of human face based on an improved seeds-growing algorithm[J]. Machine Vision and Applications(S1432-1769), 2011, 22(5): 879-887.
[54] Huang C, Hu W, Zhang Y.Robust 3D human face reconstruction by consumer binocular-stereo cameras[C]//Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry. 2012, ACM: Singapore, Singapore. 2012: 271-278.
[55] Kemelmacher-Shlizerman I, R Basri. 3D Face Reconstruction from a Single Image Using a Single Reference Face Shape[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (S0162-8828), 2011, 33(2): 394-405.
[56] S F DARPA. Human-ID 3D Face Database[DB/OL]. [2018-05-16]. http://www.csee.usf.edu/sarkar.
[57] Wu T, Zhou F, Liao Q.A fast 3D face reconstruction method from a single image using adjustable model[C]// 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Shanghai, China: IEEE, 2016.
[58] Hassner T.Viewing Real-World Faces in 3D[C]//2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE, 2013.
[59] Booth J, Antonakos E, Ploumpis S, et al.3D Face Morphable Models "In-the-Wild"[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017.
[60] Dou P, Shah S K, Kakadiaris I A.End-to-End 3D Face Reconstruction with Deep Neural Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017.
[61] Jackson A S, Bulat A, Argyriou V, et al.Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression[C]//2017 IEEE International Conference on Computer Vision (ICCV), 2017.
[62] Tewari A, Zollhofer M, Kim H, et al.MoFA: Model- Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction[C]// 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). Venice, Italy: IEEE, 2017.
[63] Tran A T, Hassner T, Masi I, et al.Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017.
[64] Taigman Y, Yang M, Ranzato M, et al.DeepFace: Closing the Gap to Human-Level Performance in Face Verification[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA: IEEE, 2014.
[65] Richardson E, Sela M, Kimmel R.3D Face Reconstruction by Learning from Synthetic Data[C]// 2016 Fourth International Conference on 3D Vision (3DV). Stanford, CA, USA: IEEE, 2016.
[66] Richardson E, Sela M, Or-EI R, et al.Learning Detailed Face Reconstruction from a Single Image[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017.
[67] Paysan P, Knothe R, Amberg B, et al.A 3D Face Model for Pose and Illumination Invariant Face Recognition[C]//2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance. Genova, Italy: IEEE, 2009.
[68] Sela M, Richardson E, Kimmel R.Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation[C]//2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017.
[69] Kemelmacher-Shlizerman I, S M Seitz. Face reconstruction in the wild[C]//2011 International Conference on Computer Vision. Barcelona, Spain: IEEE, 2011.
[70] Roth J, Yiying T, Liu X.Unconstrained 3D face reconstruction[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015.
[71] Shi F, Wu H T,Tong X, et al.Automatic acquisition of high-fidelity facial performances using monocular videos[J]. ACM Trans. Graph.(S0730-0301), 2014. 33(6): 1-13.
[72] Suwajanakorn S, I Kemelmacher-Shlizerman, S M Seitz. Total Moving Face Reconstruction[C]//European Conference on Computer Vision. Zurich, Switzerland: Springer, Cham, 2014.
[73] Cao C, Weng Y, Lin S, et al.3D shape regression for real-time facial animation[J]. ACM Trans. Graph.(S0730-0301), 2013, 32(4): 1-10.
[74] Cao C, Hou Q, Zhou K.Displaced dynamic expression regression for real-time facial tracking and animation[J]. ACM Trans. Graph.(S0730-0301), 2014, 33(4): 1-10.
[75] Cao C, Bradley D, Zhou K, et al.Real-time high-fidelity facial performance capture[J]. ACM Trans. Graph.(S0730-0301), 2015, 34(4): 1-9.
[76] Garrido P, Zollhofer M, Casas D, et al.Reconstruction of Personalized 3D Face Rigs from Monocular Video[J]. ACM Trans. Graph.(S0730-0301), 2016, 35(3): 1-15.
[77] Wu C, Bradley D, Gross M, et al.An anatomically- constrained local deformation model for monocular face capture[J]. ACM Trans. Graph.(S0730-0301), 2016, 35(4): 1-12.
[78] Ichim A E, Bouaziz S, Pauly M.Dynamic 3D avatar creation from hand-held video input[J]. ACM Trans. Graph.(S0730-0301), 2015, 34(4): 1-14.
[79] Yu R, Russell C, Campbell N D F, et al. Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video[C]//2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile:IEEE, 2015.
[80] Thies J, Zollhofer M, Stamminger M, et al.Face2Face: Real-Time Face Capture and Reenactment of RGB Videos[C]/2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016.
[81] Piotraschke M, Blanz V.Automated 3D Face Reconstruction from Multiple Images Using Quality Measures[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016.