Papers by Ehsan Kamalloo
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing kNN-based augmentation techniques blindly incorporate all samples, but MiniMax-kNN uses a subset of augmented samples to maximize KL-divergence between teacher and student models. |
| Approach: | They propose a semi-supervised approach to augmented data augmentation using kNN. |
| Outcome: | The proposed method outperforms existing kNN-based augmentation techniques on several classification tasks and requires fewer augmented examples and less computation to achieve superior performance. |
Evaluating Coherence in Dialogue Systems using Entailment (N19-1)
Copied to clipboard
| Challenge: | Evaluating open-domain dialogue systems is difficult due to the diversity of possible correct answers. |
| Approach: | They propose a set of metrics for evaluating topic coherence using distributed sentence representations and calculable approximations of human judgment using conversational coherency. |
| Outcome: | The proposed metrics can be used as a surrogate for human judgment based on conversational coherence on large-scale datasets and provide an unbiased estimate for the quality of the responses. |
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing DA methods naively add a certain number of augmented samples without considering the quality and the added computational cost of these samples. |
| Approach: | They propose a data-augmented DA technique that generates or reweights augmented samples . they say it is faster to train and can be plugged into any DA method . |
| Outcome: | The proposed technique is faster to train and more efficient than existing methods. |
FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (2022.tacl-1)
Copied to clipboard
| Challenge: | a new benchmark for hallucination-free dialogues is based on knowledge-based conversational models that generate unsupported utterances . a recent study shows that models that are trustworthy generate unverifiable or factually incorrect statements . |
| Approach: | They propose a data-centric solution to edit hallucinated responses in the Wizard of Wikipedia benchmark. |
| Outcome: | The proposed model improves on the Wizard of Wikipedia benchmark while maintaining engaging conversations. |
“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Nandan Thakur, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, Jimmy Lin
| Challenge: | Prior work on RAG grounds Large Language Models to reduce factual hallucinations lacks a comprehensive evaluation of different language families. |
| Approach: | They propose a human-annotated dataset for evaluating LLM robustness in RAG . they find that most models struggle to balance the two capacities . |
| Outcome: | The proposed dataset includes both a non-relevant and a relevant subset. |
MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages (2023.tacl-1)
Copied to clipboard
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, Jimmy Lin
| Challenge: | MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over three billion native speakers around the world. |
| Approach: | They have gathered over 726k high-quality relevance judgments for 78k queries over Wikipedia in these languages, where all annotations have been performed by native speakers hired by their team. |
| Outcome: | MIRACL covers languages that are typologically close as well as distant from 10 language families and 13 sub-families, associated with varying amounts of publicly available resources. |
Evaluating Embedding APIs for Information Retrieval (2023.acl-industry)
Copied to clipboard
Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-hermelo, Mehdi Rezagholizadeh, Jimmy Lin
| Challenge: | a growing number of language models are limiting their access to the community . we evaluate existing APIs for domain generalization and multilingual retrieval . |
| Approach: | They evaluate semantic embedding APIs in retrieval scenarios to assess their capabilities . they use BEIR and MIRACL to re-rank BM25 results using the APIs . |
| Outcome: | The proposed model is based on semantic embedding APIs that build vector representations of a given text. |
Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation models fail to identify lexical matching failures for open-domain question answering. |
| Approach: | They manually evaluate open-domain QA models by manually evaluating their answers on a popular benchmark. |
| Outcome: | The proposed model performs better on NQ-open than existing models and more than 50% of lexical matching failures are attributed to semantically equivalent answers. |