Papers by Isabel Trancoso
Automatic In-the-wild Dataset Annotation with Deep Generalized Multiple Instance Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to label large datasets that resemble real life situations are prohibitive due to the cost of manual labeling. |
| Approach: | They propose to automate the annotation process by using end-to-end differentiable neural networks to label large datasets that resemble real life conditions. |
| Outcome: | The proposed method can label a large dataset in the wild without human intervention without any cost. |
Towards Speaker Verification for Crowdsourced Speech Collections (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to detect low quality work do not address the correctness of the data. |
| Approach: | They propose an unsupervised method for measuring speaker metadata plausibility of a collection, i.e., evaluating the match (or lack thereof) between contributors and speakers. |
| Outcome: | The proposed method shows high precision in automatically classifying contributor alignment (>0.94). |
Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current evaluation practices of open domain dialogue systems are still highly dependent on human evaluation. |
| Approach: | They propose to use an annotated dataset to evaluate chatbots using large language models. |
| Outcome: | The proposed model improves over few-shot inferences on a GPT-3.5 generated dialogue dataset. |
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing meta-evaluation benchmarks are static, outdated, and lacking in multilingual coverage. |
| Approach: | They propose a framework for curating more representative open-domain dialogue evaluation benchmarks . they leverage several LLMs to generate user-chatbot multilingual dialogues conditioned on varied seed contexts based on a state-of-the-art LLM . |
| Outcome: | The proposed framework exploits state-of-the-art LLMs to perform multilingual evaluations of open-domain chatbots. |