Papers by Fernando Diaz
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to metric meta-evaluation focus on general statements about absolute and relative quality of metrics across arbitrary system outputs, but in practice, metrics are applied in highly contextual settings. |
| Approach: | They propose a method for contextual metric meta-evaluation by comparing local metric accuracy. |
| Outcome: | The proposed method compares the local metric accuracy of evaluation metrics across translation, speech recognition, and ranking tasks. |
MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Different retrievers offer distinct, often complementary signals, but they are not optimal for all queries. |
| Approach: | They propose a zero-shot, weighted combination of heterogeneous retrievers . they validate this intuition by incorporating specialized non-oracle human information sources . |
| Outcome: | Experiments show that a mixture of heterogeneous retrievers outperforms each retriever and larger models by +10.8% and +3.9% on average. |
Extrinsic Evaluation of Cultural Competence in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior studies have evaluated models’ knowledge of cultural norms, values, and artefacts, without considering how this knowledge manifests in downstream applications. |
| Approach: | They propose to evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in prompts. |
| Outcome: | The outputs of two open-ended question answering and story generation tasks are evaluated when an explicit cue of culture, specifically nationality, is perturbed in the prompts. |