Hizketa-ezagutzan oinarritutako estrategiak, euskarazko online OBHI (Ordenagailu Bidezko Hizkuntza Ikaskuntza) sistemetarako
- Autores: Igor Odriozola Sustaeta
- Directores de la Tesis: Eva Navas Cordón (dir. tes.), Inmaculada Hernáez Rioja (dir. tes.)
- Lectura: En la Universidad del País Vasco - Euskal Herriko Unibertsitatea ( España ) en 2019
- Idioma: euskera
- Títulos paralelos:
- Speech recognition based strategies for on-line Computer Assisted Language Learning (CALL) systems in Basque
- Tribunal Calificador de la Tesis: Alfonso Ortega Giménez (presid.), Ibon Saratxaga Couceiro (secret.), Alberto Abad Gareta (voc.)
- Programa de doctorado: Programa de Doctorado en Análisis y Procesamiento del Lenguaje por la Universidad del País Vasco/Euskal Herriko Unibertsitatea
- Menciones:
- Internacional
- Calificación: Sobresaliente Cum Laude
- Materias:
- Enlaces
- Tesis en acceso abierto en: ADDI
- Resumen
- English
There is a growing interest in the use of speech technology in computer-assisted language learning (CALL) systems. This is mainly due to the fact that speech technologies have considerably improved during the last years, and nowadays more and more people make use of them, ever more naturally. Literature shows that two major points of interest are involved in the use of Automatic Speech Recognition (ASR) in CALL: Computer-assisted Pronunciation Training (CAPT) and Spoken Grammar Practise (SGP). In a CAPT typical application, the user is required to read and record a sentence, and send it to the learning application server. It returns a result, generally using a three-level colour system indicating which phone has been correctly pronounced and which has not. SGP applications are not very popular yet, but some examples are multiple-choice tests, where users orally choose an answer between several choices, or even the Intelligent Language Tutoring Systems, where learners respond to cues given by the computer and the system provides feedback. Such tools can be used to strengthen the students' autonomy on their learning process, giving the opportunity of using voice to improve their pronunciation or to do grammar exercises outside the classroom.
In this work, two applications have been considered: on the one hand, a classical CAPT system, where the student records a predefined sentence and the system gives a feedback about the pronunciation; on the other hand, a novel Word-by-Word Sentence Verification (WWSV) system, where a sentence is verified sequentially, word by word, and as soon as a word is detected it is displayed to the user. The WWSV tool gives the option of creating a tool to solve grammar exercises orally (SGP). Both systems rely on utterance verification techniques, as the popular Goodness-of-pronunciation (GOP) score.
The acoustic database chosen to train the systems is the Basque Speecon-like database, the only one publicly available for Basque, specifically designed for speech recognition and recorded through microphone. This database presents several drawbacks, such as the lack of a pronunciation lexicon and some annotation files. Furthermore, it contains much dialectal speech, mainly in the free spontaneous part. In ASR, phonetic variations can be modelled using a single acoustic model. However, CAPT systems require "clean" models to use as reference. Thus, some work had to be carried out on the annotation files of the database. A number of transcriptions have been changed and a new lexicon has been created considering different dialectal alternatives. It is noticeable that the new lexicon contains in average 4.12 different pronunciations per word.
The speech recognition software used in this thesis is AhoSR. It has been created and developed by the Aholab research group, and it has been designed to cope with different recognitions tasks. In this thesis, utterance verification techniques have been implemented to be run together with the basic tasks. To do so, a parallel graph has been implemented to obtain GOP scores. For CAPT and WWSV tasks, specific search graphs have been added, in order to adapt to the needs of each of them. In addition, sockets have been implemented in the audio-input module of AhoSR. This allows real time performing when accessing the recogniser through the internet, and so it gives us the opportunity for AhoSR to be installed on a server, with universal access.
Different ways to train Hidden Markov Models (HMM) have been analysed in depth. Initially, HMMs of better quality were expected by means of using the new dictionary with alternatives. However, results do not show this, probably because of the big amount of alternative pronunciations. The addition of some manually corrected data (15 % of the training set) allows obtaining similar results to those obtained using a single-entry dictionary. In order to take advantage of the manually corrected transcriptions, different ways of training HMMs have been analysed. Thus, we have found that slightly better HMMs are achieved using data with few transcription errors in the initial stages of the training and then using the whole database.
To build the initial system, two GOP distributions were considered necessary to classify each phone: the distribution of the correctly pronounced phones and the distribution of the incorrectly pronounced ones. The GOPs of the incorrectly pronounced phones were obtained simulating errors and obtaining the GOP scores by forced alignment in the AhoSR decoder. Thus, the thresholds between correctly and incorrectly uttered phones were calculated as the Equal Error Rate (EER) point of both distributions. This approach was implemented in an initial prototype, and several laboratory experiments were performed which produced very good results. Then, the system was tested in more realistic environments: Basque language schools, among 20 students. The objective results along with the survey filled in by the 20 students who tested the system were really promising.
The initial prototype was executed locally, and we felt the need of developing a more universal system in order to be accessed from any device and anywhere. Thus, we took advantage of the specifications of the recent HTML5 standard, which let the browser access the audio input, regardless of the platform, by means of the audio API. This has given us the opportunity to create a system accessible from any operative system. Moreover, for the WWSV-based SGP task, another API of HTML5 has been used (the web API ), which creates socket-like connections between the browser and the server, in order to send audio data on the fly.
Several drawbacks have been managed for the on-line implementation of the system: for example, due to the different devices that users will use to pick up audio, some kind of parameter normalisation is needed. Furthermore, an on-line normalisation technique is necessary, since in WWSV continuous feedback must be provided before the whole signal has arrived to the recogniser. Different techniques have been tested to implement Cepstral Mean and Variance Normalisation (CMVN) and estimate the initial values of cepstral means and variances. The best results have been obtained by a hybrid approach proposed in this work, so that the initial means are estimated using the first N frames, and initial variances are obtained from the training datasets.
In addition, a new CMVN technique has been devised in this thesis: the Multi-Normalisation Scoring (MNS) based CMVN. MNS consists in generating multiple observation likelihood scores by normalising the incoming Mel-Frequency Cepstral Coefficients (MFCC) using means and variances computed from different speech datasets recorded under different conditions. The MNS-based CMVN consists in computing the probabilities of a frame to belong to different training datasets; thus, these probabilities can be used as weights to calculate an estimation of the actual means and variances. The results obtained are remarkable, mainly for clean signals. The greatest advantage of using MNS is that the CMVN can perform on-line, frame by frame, with no need to analyse the neighbouring frames or the frames of a segment to which it belongs.
Using the same MSN method, a novel and effective on-line Voice Activity Detector (VAD) has been devised as well. In a validation experiment, comparing the results of our MNS-based VAD with the results of two ITU-T VAD algorithms (G.720.1 and G.729b), we have obtained better overall results, since the classification errors are considerably lower for non-speech frames, and are comparable for speech frames. This makes our system useful for systems that require low speech error rates and also for low non-speech error rates.
Finally, Neural Networks have been used in an attempt to see the impact of different parameters at the time of training a classifier. As a consequence, we have seen that GOP scores are the most efficient parameters among durations and log-likelihoods of the previous, current and posterior phones. The results of the experiments are coherent with those obtained in the initial system.
- euskara
Ordenagailu Bidezko Hizkuntza Ikaskuntza (OBHI) sistemetan hizketa-teknologiak erabiltzeko interesa gero eta handiagoa da. Horren zergatia, hein handi batean, hizketa-teknologiek azken urteotan izan duten hobekuntza nabarmena da, eta gaur egun gero eta jende gehiagok erabiltzen ditu gero eta modu naturalagoan. Literaturak erakusten du OBHIn bi interesgune nagusi daudela hizketa-ezagutze automatikoarekin (ASR, Automatic Speech Recognition) loturik: Ordenagailu Bidezko Ebakera Lanketa (OBEL) eta Ahozko Gramatika Praktika (AGP). Ohiko OBEL aplikazio batean, esaldi bat irakurri eta grabatzeko eta ikasteko aplikazioaren zerbitzarira bidaltzeko eskatzen zaio erabiltzaileari. Zerbitzariak emaitza bat bueltatzen du, eskuarki hiru mailako kolore-sistema bat erabiliz, adierazten duena zer fonema ahoskatu den zuzen eta zein ez. AGP aplikazioak ez dira oraindik oso ezagunak, baina, besteak beste, aukera anitzeko testak daude, non erabiltzaileek ahoz hautatzen baitute erantzun bat hainbat aukeren artean; hizkuntzetarako tutoretza-sistema adimendunak ere badaude, non ikasleek ordenagailuak egindako galderei erantzun eta sistemak feedbacka ematen baitie. Halako tresnak ikasleen ikasketa-prozesuarekiko autonomia indartzeko erabil daitezke, aukera ematen baitute ahotsa erabiliz ebakera hobetzeko edo ikasgelatik kanpo gramatika-ariketak egiteko.
Lan honetan, bi aplikazio kontsideratu dira: batetik, OBEL sistema klasikoa, non aurrez zehaztutako esaldi bat grabatzen baitu ikasleak, eta sistemak ebakerari buruzko feedbacka ematen baitio; bestetik, Hitzez Hitzeko Esaldi Egiaztapenerako (HHEE) sistema berri bat, non esaldi bat sekuentzialki egiaztatzen baita, hitzez hitz, eta antzeman bezain laster erakusten baitzaio hitza erabiltzaileari. HHEE tresnarekin, gramatika-ariketak ahoz ebazteko tresna bat (AGP) eraiki daiteke. Sistema bien oinarrian, esakuntza egiaztatzeko teknikak daude, hala nola ebakera-egokitasun (GOP, Goodness Of Pronunciation) puntuazio ezaguna.
Sistema entrenatzeko hautatutako datu-base akustikoa Basque Speecon-like datu-basea da, euskaraz publikoki erabilgarri dagoen bakarra, hizketa-ezagutzarako berariaz diseinatua eta mikrofono bidez grabatua. Datu-base horrek zenbait eragozpen ditu; adibidez, ebakera-lexikoia eta zenbait etiketa-fitxategi ditu faltan. Gainera, hizketa dialektal asko du, batez ere bat-bateko atalean. ASRn, aldaera fonetikoak eredu akustiko bakar batez modela daitezke; hala ere, OBEL sistemek eredu "garbiak" behar dituzte erreferentzia gisa erabil daitezen. Horrenbestez, zenbait lan egin behar izan da datu-baseko etiketa-fitxategietan. Transkripzio asko aldatu egin behar izan dira, eta lexikoi berri bat sortu da ebakera dialektal alternatiboak kontuan hartuta. Azpimarratzekoa da lexikoi berriak 4.12 ebakera desberdin dituela, batez beste, hitz bakoitzeko.
Tesi honetan hizketa ezagutzeko erabili den softwarea AhoSR da. Aholab ikerketa-taldean sortu eta garatu da, eta ezagutze-ataza desberdinekin jarduteko diseinatu da. Tesi honetako lana egin ahal izateko, esakuntza egiaztatzeko teknikak inplementatu zaizkio, oinarrizko atazekin batera exekutatzeko. Horretarako, grafo paralelo bat inplementatu da GOP puntuazioak kalkulatzeko. OBEL eta HHEE atazetarako, berariazko bilaketa-grafoak ere erantsi zaizkio, ataza bakoitzak berezko ezaugarriak ditu eta. Gainera, socketak ere inplementatu dira AhoSRko audio-sarreraren moduluan. Horrekin, denbora errealeko funtzionamendua lortzen da ezagutzailea Internet bidez atzitzean, eta horrek AhoSR zerbitzari batean instalatzeko aukera ematen du, atzipen unibertsala izateko.
Markoven ezkutuko ereduak (HMM, Hidden Markov Models) entrenatzeko modu desberdinak aztertu dira sakonki. Hasieran, kalitate hobeko HMMak espero genituen hiztegi alternatibadun berria erabilita. Hala ere, emaitzek ez dute hori erakusten, ebakera alternatiboen hain kopuru handiagatik. Eskuz zuzendutako datuak ere erantsi dira (entrenamenduko datuen % 15), alternatibak entrenamenduan hobeki lerrokatuko zirelakoan, baina sarrera bakarreko hiztegia erabiliz lortutako emaitzen antzekoak lortzen dira. Horrenbestez, eskuz zuzendutako transkripzioak ustiatzeko, HMMak entrenatzeko modu desberdinak aztertu dira. Hala, ikusi dugu ezen HMM pittin bat hobeak lortzen dira lehendabiziko etapetan transkripzio-akats gutxiko datuak eta, gero, datu-base osoa erabilita.
Hasierako sistema eraikitzeko, kontsideratu zen bi GOP banaketa behar zirela fonema bakoitza sailkatzeko: zuzen ebakitako fonemen banaketa eta oker ebakitakoen banaketa. Oker ebakitako fonemen GOPak lortzeko, transkripzioetan akats simulatuak txertatu ziren, eta transkripziook AhoSR deskodetzailetik pasarazi ziren lerrokatze behartu moduan. Ondoren, zuzen eta oker ebakitako fonemen atalaseak kalkulatu ziren, bi banaketen errore berdineko tasaren (EER, Equal Error Rate) puntu gisa. Metodo hori hasierako prototipo batean inplementatu zen, eta laborategiko zenbait esperimentu egin ziren, oso emaitza onak izan zituztenak. Orduan, ingurune errealistago batean testatu zen sistema: euskara-ikastegi edo euskaltegietan, 20 ikasleren artean. Emaitza objektiboak eta 20 ikasleek betetako inkestaren emaitzak benetan itxaropentsuak izan ziren.
Hasierako prototipoa lokalki exekutatzen zen, eta sistema unibertsalago baten beharra sumatu genuen, edozein gailutatik eta edonondik atzitzeko. Hala, HTML5 berriaren espezifikazioak baliatu genituen, nabigatzaileak, audio APIaren bitartez, audio-sarrera atzi baitezake, plataforma edozein delarik ere. Gure sistema edozein sistema eragiletatik atzitu ahal izateko aukera eman digu horrek. Bestalde, HHEEan oinarritutako AGP atazarako, HTML5eko beste API bat erabili da (web APIa), nabigatzailearen eta zerbitzariaren artean socket moduko konexioak sortzeko aukera ematen duena, audio-datuak grabatu ahala bidaltzeko.
Sistema online inplementatzeko, zenbait eragozpenekin egin dugu topo: adibidez, erabiltzaileek audioa jasotzeko erabiliko dituzten gailuen arteko desberdintasunak direla eta, nolabaiteko parametro-normalizazioa behar da. Are gehiago, online normalizazioko teknika bat behar da, HHEEn berehalako feedbacka eman behar baita seinale osoa ezagutzailera iritsi aurretik. Hainbat teknika probatu dira batezbesteko- eta bariantza-normalizazio cepstrala (CMVN, Cepstral Mean and Variance Normalisation) inplementatzeko eta batezbestekoen eta bariantzen hasierako balioak estimatzeko. Emaitzarik onenak lan honetan proposatutako metodo hibridoaz lortu dira, non batezbestekoen hasierako balioak lehen ? bilbeak erabiliz estimatzen diren eta bariantzen hasierako balioak entrenamenduko datu-multzotik erauzten diren.
Dena dela, CMVN teknika berri bat asmatu da tesi honetan: normalizazio anitzeko puntuatzea (MNS, Multi Normalisation Scoring) metodoan oinarritutako CMVN. MNSren funtsa da hainbat behaketa-egiantz puntuazio sortzea, sarrerako Mel maiztasuneko koefiziente cepstralak (MFCC, Mel-Frequency Cepstral Coefficient) baldintza desberdinetan grabatutako hizketa datu-multzo desberdinetatik kalkulatutako batezbestekoak eta bariantzak normalizatuz. MNSan oinarritutako CMVNa honetan datza: kalkulatzen da bilbe batek entrenamenduko datu-multzo bakoitzekoa izateko zer probabilitate duen; probabilitate horiek haztapen gisa erabil daitezke batezbestekoen eta bariantzen estimazioa kalkulatzeko. Lortu ditugun emaitzak nabarmentzekoak dira, batez ere seinale garbietarako. MNS erabiltzearen abantailarik handiena da CMVNa online erabil daitekeela, bilbez bilbe, inguruko bilbeak edo uneko bilbeari dagokion segmentuko bilbeak aztertu beharrik gabe.
MSN metodo bera erabiliz, onlineko ahots-aktibitatearen detektagailu (VAD, Voice Activity Detector) berri eta eraginkor bat ere proposatu dugu. Baliozkotze-esperimentu batean, MNSan oinarritutako gure VADaren emaitzak eta ITU-Tko bi VAD algoritmo estandar (G.720.1 eta G.729b) alderatuz, emaitza orokor hobeak lortu ditugu, sailkapen-erroreak nabarmen txikiagoak baitira isiltasun-bilbeetarako, eta, antzekoak, berriz, hizketa-bilbeetarako. Gure sistema erabilgarria da, beraz, bai hizketan errore-tasa baxuak behar dituzten aplikazioetarako, bai isiltasunean errore-tasa baxuak behar dituztenetarako.
Azkenik, neurona-sareak (NN, Neural Network) erabili dira, sailkatzaile bat entrenatzeko unean parametro desberdinek duten eragina ikusteko asmoz. Horren ondorio gisa, ikusi dugu GOP puntuazioak direla parametrorik eraginkorrenak, aurreko, uneko eta ondorengo fonemen iraupenen eta log-egiantzen artean. Esperimentuen emaitzak koherenteak dira hasierako sistemaz lortutakoekin.
- English
