Papers by Yashar Mehdad
Conversational Semantic Parsing (2020.emnlp-main)
Copied to clipboard
Armen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick, Michael Haeger, Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, Sonal Gupta
| Challenge: | Structured representations for task-oriented assistant systems are limited due to the limitations of the representation. |
| Approach: | They propose a semantic representation for task-oriented conversational systems that can represent co-reference and context carryover. |
| Outcome: | The proposed model improves the best results on ATIS, SNIPS, TOP and DSTC2 by up to 5 points for slot-carryover. |
Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations (2021.findings-acl)
Copied to clipboard
| Challenge: | despite interest in explainable AI, there is increasing skepticism as to whether explanations are useful to end-users in downstream applications. |
| Approach: | They conduct user studies to measure whether explanations help users decide when to accept or reject an ODQA system's answer. |
| Outcome: | The proposed study shows that explanations outperform baselines across modalities but the best strategy varies with the modality. |
FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained sequence to sequence models are effective in making and generating NL explanations, but they have many shortcomings. |
| Approach: | They propose a model that uses sentence markers to eliminate explanation fabrication . they use fusion-in-decoder architecture to handle long input contexts . |
| Outcome: | The proposed model significantly improves on the ERASER explainability benchmark. |
Efficient One-Pass End-to-End Entity Linking for Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for entity linking are limited to entity disambiguation and require mention boundaries to be given in the input. |
| Approach: | They propose a fast end-to-end entity linking model that uses a biencoder to jointly detect mentions and link in one pass. |
| Outcome: | The proposed model outperforms the current state of the art on WebQSP and GraphQuestions with extended annotations that cover multiple entities per question. |
Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have enabled several approaches to successfully parse more complex queries, but these models require a large amount of annotated training data to parser on new domains (e.g. reminder, music). |
| Approach: | They propose a method that adapts task-oriented semantic parsers to low-resource domains and outperforms a supervised neural model at a 10-fold data reduction. |
| Outcome: | The proposed method outperforms baseline methods on a newly collected multi-domain task-oriented semantic parsing dataset (TOPv2) . |
Syntax-augmented Multilingual BERT for Cross-lingual Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained multilingual text encoders capture language syntax, helping cross-lingual transfer. |
| Approach: | They provide language syntax and train mBERT to encode universal dependency tree structure. |
| Outcome: | The proposed model improves cross-lingual transfer on PAWS-X and MLQA benchmarks by 1.4 and 1.6 points on average across all languages. |
Incremental Summarization for Customer Support via Progressive Note-Taking and Agent Feedback (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Using a mixtral-87B model, we reduce the time it takes to generate concise notes during conversations, and reduce the amount of time spent on manual review. |
| Approach: | They propose a system that combines a Mixtral-87B model for continuous note generation with a DeBERTa-based classifier to filter trivial content. |
| Outcome: | The proposed system achieved a 3% reduction in case handling time compared to bulk summarization, and high agent satisfaction ratings from surveys. |
Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support (2025.emnlp-industry)
Copied to clipboard
Cen Zhao, Tiantian Zhang, Hanchen Su, Yufeng Zhang, Shaowei Su, Mingzhi Xu, Yu Liu, Wei Han, Jeremy Werner, Claire Na Cheng, Yashar Mehdad
| Challenge: | Existing offline approaches to improve an LLM-based customer support system rely on batch annotations. |
| Approach: | They propose an agent-in-the-loop framework that integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. |
| Outcome: | The proposed framework reduces retraining cycles from months to weeks by integrating four key types of annotations directly into live customer operations. |
RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) achieve high recall amongst top few predictions, but low overall accuracy, motivating the need for answer re-ranking. |
| Approach: | They propose a method to make answer re-ranking successful for span-extraction tasks even beyond large pre-training. |
| Outcome: | The proposed approach achieves 45.5% Exact Match accuracy on Natural Questions and 61.7% on TriviaQA. |
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2021.naacl-main)
Copied to clipboard
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, Yashar Mehdad
| Challenge: | Recent advances in summarization are driven by the availability of large datasets such as the CNN-DailyMail corpus and the New York Times corpus. |
| Approach: | They propose a method for fine-tuning pretrained models for summarization in unsupervised manner . they use Wikipedia data to produce pseudo-summaries which contain characteristics of target dataset . |
| Outcome: | The proposed method achieves state-of-the-art, zero-shot abstractive summarization performance on CNN-DailyMail dataset and compares with other methods on other datasets. |
CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval (2023.acl-long)
Copied to clipboard
Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
| Challenge: | Existing multi-vector retrieval methods are slower and require more space to store indices compared to their single-vektor counterparts. |
| Approach: | They propose a multi-vector retrieval method that uses dynamic lexical routing to route different token vectors to the predicted lexicals. |
| Outcome: | The proposed method achieves state-of-the-art performance on several benchmark datasets while being nearly 40 times faster than the current state-out-of the-art method. |
MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing datasets for task-oriented dialog systems are limited and expensive . current models are based on the simple intent and slot detection paradigm for non-compositional queries. |
| Approach: | They propose to use a multilingual dataset to scale semantic parsing models to new languages . they demonstrate an average improvement of +6.3 points on Slot F1 for existing datasets . |
| Outcome: | The proposed model achieves an average improvement of +6.3 points on Slot F1 over existing models. |
Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for building dense retrievers are often misaligned and do not reflect retrieval scenario at inference time. |
| Approach: | They propose a way to validate dense retrievers using a small subset of the entire corpus. |
| Outcome: | The proposed model improves top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval by 2 4 points for open-domain question answering. |
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2022.findings-emnlp)
Copied to clipboard
Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, Wen-tau Yih
| Challenge: | Existing sparse retrievers lack the ability to match salient phrases and rare entities in the query. |
| Approach: | They introduce a dense Lexical Model that can be trained to imitate a sparse one. |
| Outcome: | The proposed model outperforms sparse retrievers on a range of tasks including five question answering datasets and the MS MARCO passage retrieval. |
ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining (2021.acl-long)
Copied to clipboard
| Challenge: | Abstractive text summarization has primarily focused on modeling news articles . lack of standardized datasets for summarizing online conversations is a major problem . |
| Approach: | They propose to crowdsource four new datasets for summarizing online conversations . they incorporate argument mining through graph construction to directly model issues, viewpoints, and assertions present in a conversation. |
| Outcome: | The proposed models are compared against widely-used conversation summarization datasets and show comparable or improved results. |
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Alexander Fabbri, Haoran Li, Ziming Mao, Griffin Adams, Borui Wang, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Existing pre-trained summarization models produce text that is factually inconsistent with the input. |
| Approach: | They present a scale-based scale for Likert rating and a scoring algorithm for Best-Worst Scaling to improve crowdsourcing reliability. |
| Outcome: | The proposed model is more reliable than existing models on two news summarization datasets. |
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. |
| Approach: | They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries. |
| Outcome: | The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
Adapting Pretrained Text-to-Text Models for Long Text Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing short-context models are limited in their domain coverage and can be used for long-sequence inputs. |
| Approach: | They propose to replace full attention in transformers with pooling-augmented blockwise attention and pretrain the model with a masked-span prediction task with spans of varying lengths. |
| Outcome: | The proposed model outperforms existing models on long-sequence summarization tasks and achieves competitive performance on long document corpora. |
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2022.findings-naacl)
Copied to clipboard
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, Scott Yih
| Challenge: | a recent study aims to answer factual questions using a structured knowledge base (KBQA). |
| Approach: | They propose a unifying approach that homogenizes all knowledge sources by reducing them to text . they demonstrate that UniK-QA is a simple and yet effective way to combine heterogeneous sources of knowledge. |
| Outcome: | The proposed approach improves state-of-the-art results on knowledge-base QA tasks by 11 points compared to graph-based methods. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
Simple Local Attentions Remain Competitive for Long-Context Tasks (2022.naacl-main)
Copied to clipboard
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad
| Challenge: | Existing models for NLP tasks require long text sequences beyond the length limit of pretrained models. |
| Approach: | They propose to pretrain large-size NLP models using the same long-doc corpus and fine tune them for real-world long-context tasks. |
| Outcome: | The proposed models can perform better under standard pretraining paradigms than longformer and Longformer. |
LLM-Friendly Knowledge Representation for Customer Support (2025.coling-industry)
Copied to clipboard
| Challenge: | a new approach to customer support is proposed to integrate large language models with a framework designed to navigate the complexities of Airbnb customer support operations. |
| Approach: | They propose a method for integrating Large Language Models with a framework designed to navigate the complexities of Airbnb customer support operations. |
| Outcome: | The proposed approach is cost-effective and improves customer support performance . it also allows human agents to focus on more complex issues, the authors show . |
Domain-matched Pre-training Tasks for Dense Retrieval (2022.findings-naacl)
Copied to clipboard
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, Yashar Mehdad
| Challenge: | Existing approaches to improve performance of pre-training tasks are needed. |
| Approach: | They propose to pre-train large bi-encoder models on a recently released set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting reddit conversation dataset. |
| Outcome: | The proposed model can be pre-trained on a set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting dataset of Reddit conversations. |
STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension (2022.emnlp-main)
Copied to clipboard
Borui Wang, Chengcheng Feng, Arjun Nair, Madelyn Mao, Jai Desai, Asli Celikyilmaz, Haoran Li, Yashar Mehdad, Dragomir Radev
| Challenge: | Abstractive dialogue summarization is an important standalone task in natural language processing, but no previous work has explored whether it can be used to boost an NLP system's performance on other important dialogue comprehension tasks. |
| Approach: | They propose a novel type of dialogue summarization task that decomposes and imitates the hierarchical, systematic and structured mental process that human beings usually go through when understanding and analyzing dialogues. |
| Outcome: | The proposed model improves the performance of transformer encoder language models on two important dialogue comprehension tasks. |
How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
| Challenge: | Existing techniques to improve dense retrieval suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, some argue due to the limited model capacity. |
| Approach: | They propose to use diverse queries and sources of supervision to train a generalizable DR to achieve high accuracy in both supervised and zero-shot retrieval. |
| Outcome: | The proposed DR can achieve state-of-the-art in supervised and zero-shot evaluations without increasing model size. |