Resumen de Statistical modelling of the baby face for 3D face reconstruction
In this thesis, we address the problem of reconstructing the 3D facial geometry of babies from uncalibrated 2D images. This topic has many different applications, but it is especially relevant for the early diagnosis of developmental disorders. It has been found that such disorders can often alter the facial morphology of patients, thus facial analysis may serve as a pre-screening tool. Furthermore, by recovering the 3D face from 2D images, the need for specialised machinery for obtaining a 3D facial image is avoided, which makes the diagnosis much more accessible. Unfortunately, the 3D face reconstruction problem is ill-posed: a 2D image is not sufficient to recover the 3D facial geometry, as it collapses one dimension, hence the 3D face cannot be recovered unequivocally without further information. A tool that has been widely used to resolve these ambiguities are the 3D morphable models (3DMMs), which are statistical models that encode the geometric variability present in a given population. Although there are several 3DMMs publicly available, they were built from populations that consist mainly of adults, and, in some cases, also include children, but not babies. As a consequence, these 3DMMs are not adequate to model the 3D facial geometry of babies, which differs much from that of adults, and thus they cannot represent the characteristic facial features of babies. For this reason, given the importance of 3DMMs for 3D face reconstruction, we address the construction of a 3D facial model of babies, the Baby Face Model (BabyFM), which is the first 3DMM build exclusively from babies and will be made publicly available to the research community. To achieve this, we propose a novel pipeline for the construction of a 3DMM that addresses the specific issues that arise when dealing with baby data, such as occlusions or extreme expressions. The pipeline consists of two phases. Firstly, we establish dense correspondences among the training faces using spectral methods, which provide a more robust theoretical foundation than state-of-the-art methods, and, indeed, will be shown to lead to more accurate correspondences. Secondly, we propose a data augmentation technique that reduces the effect of moderate-size training sets by combining two different sources of information: the geometric variability related to identity, and the geometric variability related to asymmetry. This data augmentation technique is integrated within a cutting-edge framework based on Gaussian processes, which provides a theoretically sound means to combine different sources of variability.
The construction of the BabyFM opens the possibility of addressing the 3D baby face reconstruction problem from the most recent trend in the literature, which is incorporating deep learning algorithms. Deep learning has shown great potential in capturing both the global facial shape and facial details more accurately than classical strategies, but requires sufficiently large training sets of 2D and 3D corresponding data, which are hard to find for adult faces, and even harder for babies. To overcome this lack of data, we create a synthetic training set by sampling from the BabyFM, following a strategy that has been widely used in the literature for adult data. With such synthetic training set, we train the first 3D face reconstruction system that specifically targets baby geometry, the BabyNet. It consists of a combination of a 3D autoencoder that learns a low-dimensional nonlinear latent space of the geometric variability of the baby face, and a 2D encoder that maps image features to the same latent space so that their 3D geometry can be reconstructed by the 3D decoder. This architecture, in combination with our synthetic training set, yields a 3D reconstruction system that recovers the 3D facial geometry of babies with remarkable results. The BabyNet outperforms complex deep learning-based approaches that have been trained with adult data and are not able to reproduce the characteristic facial features of babies, and it also improves the state-of-the-art results obtained with 3DMM fitting approaches, even when the BabyFM is used to recover the 3D face from the input images. The results of this thesis confirm the need for baby-specific approaches when aiming facial analysis at babies, and open the door to further research. On the one hand, the BabyFM allows us to generate a synthetic dataset with which to train the BabyNet, but it can also be used in many other research fields and applications. For example, it can serve as a reference of the normal morphology of babies in the detection of craniofacial dysmorphology patterns. On the other hand, the promising results obtained with the BabyNet suggest that it can be used in any of the applications of 3D face reconstruction, such as facial animation of babies in the computer graphics field or medical diagnosis of developmental disorders.
