Resumen de Quality-Driven video analysis for the improvement of foreground segmentation
Nowadays, the huge amount of available video content demands the creation of automatic systems for its understanding. In this context, the research community continuously improves the performance of these systems developing new algorithms that are methodologically evaluated in benchmarks via annotated ground-truth data. However, little interest is directed towards understanding the performance of the results when ground-truth is not available (stand-alone evaluation or quality estimation), which enables both an evaluation without costly annotation processes and an online understanding of errors that might be useful to improve results during run-time. In particular, the segmentation of objects of interest in videos or foreground segmentation is a relevant research area motivated by its variety of applications in topics such as video-surveillance or video edition. This thesis addresses tasks related to foreground segmentation that can improve its results while being independent of its internal details, background estimation from video frames and stand-alone quality estimation of foreground segmentation masks. Furthermore, it proposes a foreground segmentation improvement framework based on quality information.
In the first part of this thesis, two algorithms are proposed for both overcoming background estimation and applying it to stationary object detection. Therefore, this part starts by developing a block-level background estimation algorithm robust to stationary objects due to the combination of a temporal analysis to obtain a set of background candidates and spatial analysis to enforce smoothness constraints selecting the right background candidate in each image location. Then, a practical use of background estimation for stationary object detection is explored by continuously estimating background images at different sampling instants and comparing them to determine stationarity. This approach is based on an online clustering that enables fast adaptation to scene variations while analyzing spatio-temporal changes to detect the stationary objects. Experiments on a variety of datasets demonstrate the efficiency of the two proposed background estimation related approaches proposed.
In the second part, this thesis estimates the quality of foreground segmentation algorithms from a stand-alone perspective and proposes a post-processing framework that exploits quality information to improve algorithm results. Firstly, this part addresses the stand-alone evaluation of foreground masks by extracting properties over their connected components (blobs). In particular, an extensive comparison in terms of correlations with ground-truth based evaluation metrics and capabilities for quality-levels discrimination for 21 measures, revealing that fitness between blobs and segmented image regions (fitness-to-regions) is a good quality estimator. Afterwards, this thesis proposes a post-processing framework to improve foreground segmentation performance exploiting fitness-to-regions. To do so, a motion-aware hierarchical image segmentation of each frame is built to allow quality estimation at different degrees of detail (without merging foreground and background image regions). This hierarchical framework enables the estimation of a combined quality. Finally, this foreground quality is transformed and exploited together with spatial color relations to improve the foreground mask via an optimal labeling process. The experiments conducted over large and heterogeneous datasets with varied challenges validate the utility of this approach.
