Papers by Yassine Benajiba
Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty (2025.findings-emnlp)
Copied to clipboard
Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
| Challenge: | Existing methods to gauge model’s uncertainty through self-consistency in responses to the target query are misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same query when answering a knowledge-preserving perturbation of the query. |
| Approach: | They propose a method that uses multi-agent interaction to estimate black-box LLMs' uncertainty. |
| Outcome: | The proposed method outperforms existing self-consistency based methods and improves hallucination detection. |
Characterizing and Measuring Linguistic Dataset Drift (2023.acl-long)
Copied to clipboard
Tyler Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth
| Challenge: | Existing metrics for dataset drift have not considered specific dimensions of linguistic drift that affect model performance. |
| Approach: | They propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. |
| Outcome: | The proposed metrics are more effective than previous metrics at predicting out-of-domain model accuracies compared to popular fine-tuned embedding distances . |
Open Domain Question Answering with Conflicting Contexts (2025.findings-naacl)
Copied to clipboard
Siyi Liu, Qiang Ning, Kishaloy Halder, Zheng Qi, Wei Xiao, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, Dan Roth
| Challenge: | Open domain question answering systems often rely on information retrieved from large collections of text to answer questions. |
| Approach: | They evaluate and benchmark three powerful Large Language Models with a dataset . they find that 25% of unambiguous open domain questions can lead to conflicting contexts . |
| Outcome: | The proposed model can't be used to answer questions with conflicting contexts . it can be fine tuned to provide richer information into the model's training . |
Diable: Efficient Dialogue State Tracking as Operations on Tables (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for dialogue state tracking use the full dialogue history as input and generate the entire state from scratch at each dialogue turn. |
| Approach: | They propose a task formalisation that represents the dialogue state as a table and formalises it as 'table manipulation task' they represent the dialogue as if it were a list with all the slots and generate the entire state from scratch at each dialogue turn. |
| Outcome: | The proposed system outperforms existing systems while maintaining competitive accuracy. |
TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025.findings-acl)
Copied to clipboard
| Challenge: | Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks. |
| Approach: | They propose to augment LoCoMo dialogues and create multi-choice QAs to construct a temporal reasoning evaluation task and a framework to enhance temporal thinking capabilities of LLM-agents. |
| Outcome: | The proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via the proposed framework. |
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (2023.eacl-main)
Copied to clipboard
| Challenge: | Temporal concept drift is a problem of data changing over time. |
| Approach: | They benchmark 11 pretrained masked language models on a series of tests to evaluate temporal concept drift. |
| Outcome: | The proposed framework evaluates 11 pretrained masked language models on a series of tests . it aims to reveal how robust an MLM is over time and provide a signal in case it has become outdated . |
Aspect On: an Interactive Solution for Post-Editing the Aspect Extraction based on Online Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Experimental results show that Aspect On dramatically reduces the number of user clicks and effort required to post-edit the aspects extracted by the model. |
| Approach: | They propose an online learning-based aspect extraction solution that allows users to post-edit the aspect extraction with little effort. |
| Outcome: | The proposed solution dramatically reduces the number of user clicks and effort required to post-edit the aspects extracted by the model. |
MemInsight: Autonomous Memory Augmentation for LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM) agents have evolved to intelligently process information, make decisions, and interact with users or tools. |
| Approach: | They propose an autonomous memory augmentation approach to enhance semantic data representation and retrieval mechanisms by leveraging historical interactions. |
| Outcome: | The proposed approach outperforms a baseline RAG by 34% in recall for LoCoMo retrieval on three task scenarios and boosts persuasiveness of recommendations by 14%. |
Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Text-to-SQL technology bridges natural language (NL) questions and database querying. |
| Approach: | They propose a method for evaluating LLM-generated NLRs using Combo-Eval and a dataset for NLR benchmarking. |
| Outcome: | The proposed method reduces LLM calls by 25-61% and improves performance across scenarios with and without ground truth references. |
MEAV: Model Editing with Alignment Vectors for inference time LLM alignment in single and multidomain preference spectrum (2026.findings-acl)
Copied to clipboard
Sadat Shahriar, Zheng Qi, Nikolaos Pappas, Srikanth Doss, Kishaloy Halder, Monica Sunkara, Manuel Mager, Yassine Benajiba
| Challenge: | Existing training-time alignment methods require full retraining when a change is needed. |
| Approach: | They propose an inference-time model-editing-based alignment method that learns encoded representations of preference dimensions and allows dynamic adjusting of the model behavior. |
| Outcome: | The proposed method can be used to align large language models to human preferences . it reduces the cost of inference by half compared to the prompt engineering approach . |
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
| Challenge: | LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone. |
| Approach: | They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs. |
| Outcome: | The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs. |
Taxonomy Expansion for Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
Karthikeyan K, Yogarshi Vyas, Jie Ma, Giovanni Paolini, Neha John, Shuai Wang, Yassine Benajiba, Vittorio Castelli, Dan Roth, Miguel Ballesteros
| Challenge: | Training a Named Entity Recognition model involves fixing a taxonomy of entity types . however, requirements evolve and a model may need to recognize additional entity types. |
| Approach: | They propose a method that uses only partially annotated datasets to train a model to recognize additional entity types. |
| Outcome: | The proposed approach performs better with partially annotated datasets than other approaches . the gap between the proposed approach and other approaches is large in additional datasets . |
Aligning to Constraints for Data-Efficient Language Model Customization (2025.findings-naacl)
Copied to clipboard
Fei Wang, Chao Shang, Shuai Wang, Sarthak Jain, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, Dan Roth
| Challenge: | General-purpose language models (LMs) are aligned to diverse user intents, but fall short when it comes to specific applications. |
| Approach: | They propose a framework that uses constraints to automatically produce supervision signals for user alignment with constraints. |
| Outcome: | The proposed framework can produce supervision signals for user alignment with constraints. |
Towards Long Context Hallucination Detection (2025.findings-naacl)
Copied to clipboard
Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, Dan Roth
| Challenge: | Large language models are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. |
| Approach: | They propose a dataset specifically designed for long-context hallucination detection. |
| Outcome: | The proposed architecture outperforms existing models while providing faster inference. |
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (2026.findings-acl)
Copied to clipboard
Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi, Dan Roth
| Challenge: | Large language models suffer performance degradation when user instructions and context are distributed over multiple conversational turns. |
| Approach: | They propose a framework that condenses chat history in the background without disrupting the user experience. |
| Outcome: | The proposed framework reduces token counts by up to 72% in 10-turn dialogues while remaining robust to distractors and irrelevant turns. |
Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | generating aspect-specific and general opinion summaries is challenging due to the lack of annotated data. |
| Approach: | They propose two unsupervised approaches to generate aspect-specific and general opinion summaries by training on synthetic datasets constructed with aspect-related review contents. |
| Outcome: | The proposed method outperforms existing methods on space and Oposum+ and on other metrics. |
ODIST: Open World Classification via Distributionally Shifted Instances (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work to achieve open-world classification capability in natural language processing and computer vision focuses on decision boundary finding. |
| Approach: | They propose a method that can create out-of-domain instances from in-domain training instances with the help of a pre-trained generative language model. |
| Outcome: | The proposed method can create out-of-domain instances from the in-domain training instances with the help of a pre-trained generative language model. |
JTPRO: A Joint Tool–Prompt Reflective Optimization Framework for Language Agents (2026.findings-acl)
Copied to clipboard
Sandip Ghoshal, Anshul Mittal, Jyotika Singh, Miguel Ballesteros, Weiyi Sun, Fang Tu, Shailender Singh, Yassine Benajiba, Fahad Shah, Sujeeth Bharadwaj, Sujith Ravi, Dan Roth
| Challenge: | Large language model agents struggle with ambiguous tool descriptions and underspecified tool schemas that ignore tool-specific nuances. |
| Approach: | They propose a framework for improving tool-calling reliability in trace-supervised settings by rolling out-driven reflection. |
| Outcome: | The proposed framework outperforms baselines and reflective prompt optimizers by 5%–20% on OSR. |
Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidth (2026.eacl-long)
Copied to clipboard
Michelle Yuan, Weiyi Sun, Amir H. Rezaeian, Jyotika Singh, Sandip Ghoshal, Yao-Ting Wang, Miguel Ballesteros, Yassine Benajiba
| Challenge: | despite advances in transformers, their theoretical limitations in discrete reasoning remain a critical open problem. |
| Approach: | They synthesize recent advances from three theoretical perspectives to clarify structural and computational barriers transformers face when performing symbolic computations. |
| Outcome: | The proposed models excel at pattern matching and interpolation, but they face bottlenecks in communication and depth constraints. |