Papers by Alistair Willis
Rethinking the Agreement in Human Evaluation Tasks (C18-1)
Copied to clipboard
| Challenge: | In natural language processing, IAA is often viewed as a means of assessing the quality of data on a task, in particular, the reliability. |
| Approach: | They propose a new approach to use agreement metrics in natural language generation evaluation tasks to reduce subjective bias. |
| Outcome: | The proposed approach is based on the inter-annotator agreement (IAA) of natural language generation tasks. |
Identifying Annotator Bias: A new IRT-based method for bias identification (2020.coling-main)
Copied to clipboard
| Challenge: | An important factor that can affect the Inter Annotator Agreement (IAA) is the presence of annotator bias. |
| Approach: | They propose a new interpretation and application of the Item Response Theory to detect annotators' bias and characterise annotation disagreement. |
| Outcome: | The proposed method can be used to spot outliers, improve annotation guidelines and provide a better picture of the annotation reliability. |