Deep learning methodologies for textual and graphical content-based analysis of handwritten text images
- Autores: José Ramón Prieto Fontcuberta
- Directores de la Tesis: Enrique Vidal Ruiz (dir. tes.), Lorenzo Quirós Díaz (dir. tes.)
- Lectura: En la Universitat Politècnica de València ( España ) en 2024
- Idioma: inglés
- Tribunal Calificador de la Tesis: Thierry Paquet (presid.), Carlos David Martínez Hinarejos (secret.), Giorgos Sfikas (voc.)
- Programa de doctorado: Programa de Doctorado en Informática por la Universitat Politècnica de València
- Menciones:
- Internacional
- Calificación: Sobresaliente
- Materias:
- Enlaces
- Tesis en acceso abierto en: RiuNet
- Resumen
This thesis addresses unresolved issues in the field of Artificial Intelligence as applied to historical handwritten documents. The challenges include not only the degradation of the documents but also the scarcity of available data for training specialized models. This limitation is particularly relevant when the trend is to use large datasets and massive models to achieve significant breakthroughs.
First, we provide an overview of various techniques and concepts used throughout the thesis. Different ways of representing data are explored, including images, text, and graphs. Probabilistic Indices (PrIx) are introduced for textual representation and its encoding using TfIdf is be explained. We also discuss selecting the best input features for neural networks using Information Gain (IG). In the realm of neural networks, specific models such as Multilayer Perceptron (MLP), Convolutional Neural Networks (CNNs), and graph-based networks (GNNs) are covered, along with a brief introduction to transformers.
The first problem addressed in this thesis is the segmentation of historical handwritten books into semantic units, a complex and recurring challenge in archives worldwide. Unlike modern books, where chapter segmentation is relatively straightforward, historical books present unique challenges due to their irregularities and potential poor preservation. To the best of our knowledge, this thesis formally defines this problem. We propose a pipeline to consistently extract these semantic units in two variations: one with corpus-specific constraints and another without them.
Various types of neural networks are employed, including Convolutional Neural Networks (CNNs) for classifying different parts of the image and Region Proposal Networks (RPNs) and transformers for detecting and classifying regions. Additionally, a new metric is introduced to measure the information loss in the detection, alignment, and transcription of these semantic units. Finally, different decoding methods are compared, and the results are evaluated across up to five different datasets.
In another chapter, we tackle the challenge of classifying non-transcribed historical handwritten documents, specifically notarial deeds, from the Provincial Historical Archive of Cádiz. A framework is developed that employs Probabilistic Indices (PrIx) for classifying these documents, and this is compared to 1-best transcriptions obtained through Handwritten Text Recognition (HTR) techniques.
In addition to conventional classification within a closed set of classes (Close Set Classification, CSC), this thesis introduces the Open Set Classification (OSC) framework. This approach not only classifies documents into predefined classes but also identifies those that do not belong to any of the established classes, allowing an expert to label them.
Various techniques are compared, and two are proposed. One approach without using a threshold on the posterior probabilities generated by the neural network model. At the same time, the other employs a threshold on these probabilities, with the option for manual adjustment according to the expert's needs.
In a third chapter, this thesis focuses on Information Extraction (IE) from handwritten tabular documents. A pipeline is developed that starts with detecting text in images containing tables, line by line, followed by its transcription using HTR techniques. In parallel, various models are trained to identify the structure of the tables, including rows, columns, and header sections.
The pipeline also addresses common issues in handwritten tables, such as multi-span columns and substituting ditto marks. Additionally, a language model specifically trained to detect table headers automatically is employed.
Two datasets are used to demonstrate the effectiveness of the pipeline in the IE task, and areas for improvement within the pipeline itself are identified for future research.
