Topic unit detection in spontaneous speech: Measuring reliability using the Kappa statistic

Cavalcante, Frederico Amorim ^[2] ; Raso, Tommaso ; Bossaglia, Giulia ^[2] ; Mittmann, Maryualê ^[1] ; Rocha, Bruno ^[2]
1. [1] Universidade do Vale do Itajaí
  
  Universidade do Vale do Itajaí
  
  Brasil
2. [2] UFMG
Localización: CHIMERA: Romance Corpora and Linguistic Studies, ISSN-e 2386-2629, Nº. 7, 2020, págs. 69-106
Idioma: inglés
Enlaces
- Texto completo

Resumen
- This paper deals with an inter-annotator agreement test involving the identification of the information unit of Topic as defined within the framework of the Language into Act Theory (L-AcT). Fleiss’s kappa statistic was used to measure the agreement among the four annotators who took part in the test. The data used was sampled from C-ORAL-BRASIL II, a spontaneous speech corpus of Brazilian Portuguese. The paper begins by outlining of the theoretical underpinnings of L-AcT, dedicating special attention to aspects directly related to the notion of Topic. Section 2 presents the pilot test and discusses methodological and theoretical issues that were relevant for the design of the protocol that was eventually used in the actual test. Sections 3 and 4 deal with the test, its protocol and results (the kappa coefficient for the general agreement was 0.79, which by usual standards represents a substantial agreement). Section 5 first provides a brief review of a few studies conducted according to other frameworks which have dealt with inter-rater agreement on the annotation of information structure categories. Finally, the errors observed in the test are analyzed qualitatively.