Papers by Thórhildur Thorleiksdóttir
Dynamic Human Evaluation for Relative Model Comparisons (2022.lrec-1)
Copied to clipboard
| Challenge: | Automated metrics have reported flaws when applied to measure quality aspects of generated text and have been shown to correlate poorly with human judgements. |
| Approach: | They propose an agent-based framework to measure the required number of human annotations when evaluating generated outputs in relative comparison settings. |
| Outcome: | The proposed model can be compared with a crowdsourced case study and a simulation with simulated human judgements. |