Automatic Machine Translation Evaluation: A Qualitative Approach

• Autores: Elisabet Comelles Pujadas
• Directores de la Tesis: M. Victoria Arranz (dir. tes.)
• Lectura: En la Universitat de Barcelona ( España ) en 2015
• Idioma: catalán
• Tribunal Calificador de la Tesis: Ana María Fernández Montraveta (presid.), Isabel Verdaguer (secret.), Meritxell González Bermúdez (voc.)
• Materias:
  • Lingüística
    • Lingüística aplicada
      • Traducción automática
• Enlaces
  • Tesis en acceso abierto en:  TDX  Dipòsit Digital de la UB 
• Dialnet Métricas: 1 Cita
• Resumen

  • The present study addresses the problem of Automatic Evaluation of Machine Translation (MT) from a linguistic perspective. Most of the studies performed in this area focus on quantitative analyses based on correlation coefficients; however, little has been done as regards a more qualitative approach, going beyond correlations and analysing data in detail. This thesis aims at shedding some light on the suitability, influence and combination of linguistic information to evaluate MT output, not restricting our research to the correlation with human judgements but basing it on a qualitative analysis. More precisely, this research intends to emphasize the effectiveness of linguistic analysis in order to identify and test those linguistic features that help in evaluating traditional concepts of adequacy and fluency. In order to perform this research we have focused on MT output in English, with an application to Spanish so as to test the portability of our approach. The starting point of this work was a linguistic analysis of both MT output and reference segments with the aim of highlighting not only those linguistic errors that an automatic MT evaluation metric must identify, but also those positive linguistic features that must be taken into account, identified and treated as correct linguistic phenomena. Once the linguistic analysis was conducted and in order to confirm our hypotheses and check whether those linguistic phenomena and traits identified in the analysis were helpful to evaluate MT output, we designed and implemented a linguistically-motivated MT metric, VERTa, to evaluate English output. Several experiments were conducted with this first version of VERTa in order to test the suitability of the linguistic features selected and how they should be combined so as to evaluate fluency and adequacy separately. Besides using information provided by correlations as a guide we also performed a detailed analysis of the metric’s output every time linguistic features were added and/or combined. After performing these experiments and checking the suitability of the linguistic information used and how it had to be used and combined, VERTa’s parameters were adjusted and an updated and optimised version of the metric was ready to be used. With this updated version and for the sake of comparison, a meta-evaluation of the metric for adequacy, fluency and MT quality was conducted, as well as a comparison to some of the best-known and widely-used MT metrics, showing that it outperformed them all when adequacy and fluency were assessed. Finally, we ported our MT metric to Spanish with the aim of studying its portability by checking which linguistic features in our metric would have to be slightly modified, which changes would have to be performed and finally if the metric would be easy to adapt to a new language. Furthermore, this version of VERTa for Spanish was compared to other well-known metrics used to evaluate Spanish, showing that it also outperformed them.