Papers by Saab Mansour
NatCS: Eliciting Natural Customer Support Dialogues (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue datasets do not reflect the expected characteristics of real customer support conversations. |
| Approach: | They propose to collect real customer service conversations from real conversations . they show that dialogue act annotations provide more effective training data . |
| Outcome: | The proposed approach is more representative of real human-to-human conversations compared to existing dialogue datasets . the proposed approach can be used to facilitate open research in natural dialog systems . |
MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization (2025.findings-emnlp)
Copied to clipboard
Yinhong Liu, Jianfeng He, Hang Su, Ruixue Lian, Yi Nian, Jake W. Vincent, Srikanth Vishnubhotla, Robinson Piramuthu, Saab Mansour
| Challenge: | Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. |
| Approach: | They propose a meta-evaluation benchmark for multimodal dialogue summarization based on image-sharing dialogues, corresponding summaries and human judgments . |
| Outcome: | The proposed framework is the first to identify and formalize key evaluation dimensions specific to MDS. |
FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis. |
| Approach: | They propose a fine-grained evaluator specifically tailored for the summarization task using large language models. |
| Outcome: | The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods. |
Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on robustness of pretrained multilingual models are limited to the English language. |
| Approach: | They propose to use data augmentation and contrastive loss term to boost robustness of multilingual models in cross-lingual settings. |
| Outcome: | The proposed model outperforms existing models on clean and noisy data in the cross-lingual setting. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |
Knowledge-Driven Slot Constraints for Goal-Oriented Dialogue Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Traditional goal-oriented dialogue systems allow execution of validation rules as a post-processing step after slots have been filled which can lead to error accumulation. |
| Approach: | They propose a task of constraint violation detection based on knowledge-driven slot constraints . they propose methods to integrate external knowledge into the system and compare it to traditional rule-based pipeline approach . |
| Outcome: | The proposed task compares to the existing system and a rule-based pipeline. |
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets (2024.naacl-long)
Copied to clipboard
Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Lijia Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
| Challenge: | Existing approaches to augment textual dialogues with retrieved images pose privacy, diversity, and quality constraints. |
| Approach: | They propose a framework to augment text-only dialogues with diverse and high-quality images by using a diffusion model and a feedback loop. |
| Outcome: | The proposed framework is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small. |
Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies suggest different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. |
| Approach: | They propose to use Optimal Transport as an alignment objective during fine-tuning to improve multilingual contextualized representations for downstream cross-lingual transfer. |
| Outcome: | The proposed method achieves better performance on two tasks (XNLI and XQuAD) and is competitive with existing methods. |
Multilingual Self-Taught Faithfulness Evaluators (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing faithfulness evaluation approaches are mostly English-focused and require expensive human-labeled training data for fine-tuning specialized models. |
| Approach: | They propose a framework that learns exclusively from synthetic multilingual data while leveraging cross-lingual transfer learning to improve an LLM's general language capabilities. |
| Outcome: | The proposed framework shows that it improves over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches. |
Active Generalized Category Discovery with Diverse LLM Feedback (2026.eacl-long)
Copied to clipboard
| Challenge: | Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled data from known categories. |
| Approach: | They propose a framework for generalized category discovery that actively learns from diverse and collaborative feedback. |
| Outcome: | The proposed framework improves instance-level contrastive features, generates category descriptions, and aligns uncertain instances with LLM-selected category descriptions. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Flow-adhering planning algorithm for task oriented dialogs (TODs) is a task-oriented dialog (TO) that can be used for task planning and API usage. |
| Approach: | They propose a Flow-Adhering Planning algorithm that follows predefined flows and preserves API dependencies in task oriented dialogs. |
| Outcome: | The proposed algorithm outperforms other decoding and prompting-based baselines in task oriented dialogs. |
CERET: Cost-Effective Extrinsic Refinement for Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate incomplete, biased or misleading outputs in their initial attempts. |
| Approach: | They propose a method for refining text generation that takes into account semantic stability, entailment and inter-sample uncertainty measures. |
| Outcome: | The proposed method outperforms self-consistency and self-rerank baselines under various task setups by 1.6% and 3.5% respectively. |
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are slow and abstractive, but we propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries. |
| Approach: | They propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries by exposing diverse pseudo summary with two supervision to the student model. |
| Outcome: | The proposed method outperforms previous methods in abstractive summarization distillation, producing highly abstractive and informative summaries. |
Nearest Neighbour Few-Shot Learning for Cross-lingual Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models can cause over-fitting when limited data are available. |
| Approach: | They propose to use a nearest-neighbor few-shot technique to improve cross-lingual adaptation using 16 distinct languages across two NLP tasks. |
| Outcome: | The proposed approach improves fine-tuning using only a handful of labeled samples in target locales and also generalizes across tasks. |
Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024.acl-long)
Copied to clipboard
| Challenge: | Conversational systems often rely on embedding models for intent classification and intent clustering tasks. |
| Approach: | They propose a toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task. |
| Outcome: | The proposed model improves on the linguistic dimensions while affecting performance on downstream task metrics. |
Eliciting Better Multilingual Structured Reasoning from LLMs through Code (2024.acl-long)
Copied to clipboard
| Challenge: | xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. |
| Approach: | They propose a multilingual structured reasoning and explanation dataset that covers four tasks across six languages and extends the English STREET benchmark to 5 additional diverse languages. |
| Outcome: | The proposed models show improved multilingual performance on scientific commonsense reasoning subtasks and no regression on non-reasoning tasks. |
End-to-End Slot Alignment and Recognition for Cross-Lingual NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to translated utterances. |
| Approach: | They propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. |
| Outcome: | The proposed model outperforms a simple label projection method on most languages and achieves competitive performance to the more complex, state-of-the-art projection method with only half the training time. |
Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection (2024.naacl-long)
Copied to clipboard
| Challenge: | Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data. |
| Approach: | They propose a scoring approach that encapsulates three primary dimensions of summarization model quality. |
| Outcome: | The proposed method reduces reliance on human-labeled data and improves the performance of summarization models. |
Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models lack domain-specific knowledge that does not naturally occur in pre-training data. |
| Approach: | They propose to inject domain-specific knowledge prior to fine-tuning on TOD tasks by using adapters that can be easily integrated with PLMs. |
| Outcome: | The proposed methods show that they can inject domain-specific knowledge prior to fine-tuning on TOD tasks. |
DeAL: Decoding-time Alignment for Large Language Models (2025.acl-long)
Copied to clipboard
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth
| Challenge: | Large Language Models (LLMs) are expected to generate content aligned with human preferences. |
| Approach: | They propose a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). |
| Outcome: | The proposed framework allows the user to customize reward functions and enables Decoding-time Alignment of LLMs. |
ODIST: Open World Classification via Distributionally Shifted Instances (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work to achieve open-world classification capability in natural language processing and computer vision focuses on decision boundary finding. |
| Approach: | They propose a method that can create out-of-domain instances from in-domain training instances with the help of a pre-trained generative language model. |
| Outcome: | The proposed method can create out-of-domain instances from the in-domain training instances with the help of a pre-trained generative language model. |
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (2025.naacl-long)
Copied to clipboard
Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, Hang Su
| Challenge: | Existing approaches to evaluate faithfulness of summaries are often fooled by the fluency of the text and struggle with identifying errors. |
| Approach: | They propose an approach to summary faithfulness evaluation where multiple LLM-based agents are assigned initial stances and forced to come up with a reason to justify belief. |
| Outcome: | The proposed approach can identify ambiguities and have even stronger performance on non-ambiguous summaries. |
Label Semantic Aware Pre-training for Few-shot Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for text classification use label semantics but few studies have attempted to give models access to informative representations of labels. |
| Approach: | They propose to use label semantics to train generative models by performing secondary pre-training on labeled sentences from a variety of domains. |
| Outcome: | The proposed approach improves generalization and data efficiency of text classification systems while maintaining comparable performance to state-of-the-art models. |
Structured List-Grounded Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Document-grounded dialogue systems aim to answer user queries by leveraging external information. |
| Approach: | They propose a dataset to evaluate QA systems' ability to interpret and use structured lists . they use language models and model-based filtering processes to enhance data quality . |
| Outcome: | The proposed model outperforms baselines on the LIST2QA dataset . it shows that the proposed model is more accurate and complete than baselines . |