Papers by Marian Marchal
Establishing Annotation Quality in Multi-label Annotations (2022.coling-1)
Copied to clipboard
| Challenge: | Multi-label annotations allow multiple interpretations of a single item, but they also affect the chance that two coders agree with each other. |
| Approach: | They propose a bootstrapped method to obtain chance agreement for each measure and a method to get an adjusted agreement coefficient that is more interpretable. |
| Outcome: | The proposed method allows for an adjusted agreement coefficient that is more interpretable on simulated datasets. |