| Title: |
Soft metrics for evaluation with disagreements: an assessment |
| Authors: |
Rizzi, Giulia; Leonardelli, Elisa; Poesio, Massimo; Uma, Alexandra; Pavlovic, Maja; Paun, Silviu; Rosso, Paolo; Fersini, Elisabetta; Sub Natural Language Processing; Natural Language Processing; Abercrombie, Gavin; Basile, Valerio; Bernardi, Davide; Dudy, Shiran; Frenda, Simona; Havens, Lucy; Tonelli, Sara |
| Publication Year: |
2024 |
| Subject Terms: |
Language and Linguistics; Education; Library and Information Sciences; Linguistics and Language |
| Description: |
The move towards preserving judgement disagreements in NLP requires the identification of adequate evaluation metrics. We identify a set of key properties that such metrics should have, and assess the extent to which natural candidates for soft evaluation such as Cross Entropy satisfy such properties. We employ a theoretical framework, supported by a visual approach, by practical examples, and by the analysis of a real case scenario. Our results indicate that Cross Entropy can result in fairly paradoxical results in some cases, whereas other measures Manhattan distance and Euclidean distance exhibit a more intuitive behavior, at least for the case of binary classification. |
| Document Type: |
book part |
| File Description: |
application/pdf |
| Language: |
English |
| Relation: |
https://dspace.library.uu.nl/handle/1874/482107 |
| Availability: |
https://dspace.library.uu.nl/handle/1874/482107 |
| Rights: |
info:eu-repo/semantics/OpenAccess |
| Accession Number: |
edsbas.3C9610BC |
| Database: |
BASE |