Semantic and generic object segmentation for scene analysis using rgb-d data
- Autores: Xiao Lin
- Directores de la Tesis: Montserrat Pardas Feliu (dir. tes.), Joan R. Casas (dir. tes.)
- Lectura: En la Universitat Politècnica de Catalunya (UPC) ( España ) en 2018
- Idioma: español
- Tribunal Calificador de la Tesis: Glòria Haro Ortega (presid.), Josep Ramon Morros Rubio (secret.), Antoni Jaume Capó (voc.)
- Programa de doctorado: Programa de Doctorado en Teoría de la Señal y Comunicaciones por la Universidad Politécnica de Catalunya
- Materias:
- Enlaces
- Tesis en acceso abierto en: TDX
- Resumen
In this thesis, we study RGB-D based segmentation problems from different perspectives in terms of the input data. Apart from the basic photometric and geometric information contained in the RGB-D data, also semantic and temporal information are usually considered in an RGB-D based segmentation system.
The first part of this thesis focuses on an RGB-D based semantic segmentation problem, where the predefined semantics and annotated training data are available. First, we review how RGB-D data has been exploited in the state of the art to help training classifiers in a semantic segmentation tasks. Inspired by these works, we follow a multi-task learning schema, where semantic segmentation and depth estimation are jointly tackled in a Convolutional Neural Network (CNN). Since semantic segmentation and depth estimation are two highly correlated tasks, approaching them jointly can be mutually beneficial. In this case, depth information along with the segmentation annotation in the training data helps better defining the target of the training process of the classifier, instead of feeding the system blindly with an extra input channel. We design a novel hybrid CNN architecture by investigating the common attributes as well as the distinction for depth estimation and semantic segmentation. The proposed architecture is tested and compared with state of the art approaches in different datasets.
Although outstanding results are achieved in semantic segmentation, the limitations in these approaches are also obvious. Semantic segmentation strongly relies on predefined semantics and a large amount of annotated data, which may not be available in more general applications. On the other hand, classical image segmentation tackles the segmentation task in a more general way. But classical approaches hardly obtain object level segmentation due to the lack of higher level knowledge. Thus, in the second part of this thesis, we focus on an RGB-D based generic instance segmentation problem where temporal information is available from the RGB-D video while no semantic information is provided. We present a novel generic segmentation approach for 3D point cloud video (stream data) thoroughly exploiting the explicit geometry and temporal correspondences in RGB-D. The proposed approach is validated and compared with state of the art generic segmentation approaches in different datasets.
Finally, in the third part of this thesis, we present a method which combines the advantages in both semantic segmentation and generic segmentation, where we discover object instances using the generic approach and model them by learning from the few discovered examples by applying the approach of semantic segmentation. To do so, we employ the one shot learning technique, which performs knowledge transfer from a generally trained model to a specific instance model. The learned instance models generate robust features in distinguishing different instances, which is fed to the generic segmentation approach to perform improved segmentation. The approach is validated with experiments conducted on a carefully selected dataset.
