Papers by Joris Baan
Stop Measuring Calibration When Humans Disagree (2022.emnlp-main)
Copied to clipboard
| Challenge: | Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., predictive probabilities are a good indication of how likely a prediction is to be correct. |
| Approach: | They propose to measure calibration to human majority given inherent disagreements on tasks where humans inherently disagree about which class applies. |
| Outcome: | The proposed measures capture key statistical properties of human judgements including class frequency, ranking and entropy. |
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? (2024.eacl-short)
Copied to clipboard
| Challenge: | In modern NLP, neural networks are the de-facto standard to predict complex probability measures from available context. |
| Approach: | They propose to use a single predictive distribution to evaluate models with disentangled representations of uncertainty about predictions and uncertainty about human labels. |
| Outcome: | The proposed models are crucial for trustworthy and fair NLP systems, but exploiting a single distribution is limiting. |
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |