Papers by Miguel Ballesteros
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |
Sequential Cross-Document Coreference Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for cross-document coreference resolution have been used for within-document entity coreference but have been relatively limited. |
| Approach: | They propose a model that extends the efficient sequential prediction paradigm for coreference resolution to cross-document settings and achieves competitive results for both entity and event coreference. |
| Outcome: | The proposed model achieves competitive results for entity and event coreference while minimizing error propagation in complex reasoning tasks. |
Multimodal Emoji Prediction (N18-2)
Copied to clipboard
| Challenge: | Emojis are small images that are commonly included in social media text messages. |
| Approach: | They propose a multimodal approach that is able to predict emojis in Instagram posts by using both text and image. |
| Outcome: | The proposed model incorporates both text and image to improve accuracy . |
Characterizing and Measuring Linguistic Dataset Drift (2023.acl-long)
Copied to clipboard
Tyler Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth
| Challenge: | Existing metrics for dataset drift have not considered specific dimensions of linguistic drift that affect model performance. |
| Approach: | They propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. |
| Outcome: | The proposed metrics are more effective than previous metrics at predicting out-of-domain model accuracies compared to popular fine-tuned embedding distances . |
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. |
| Approach: | They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks. |
| Outcome: | The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task. |
A Bag of Tricks for Dialogue Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using a pretrained sequence-to-sequence language model, we explore speaker name substitution, negation scope highlighting, multi-task learning with relevant tasks, and pretraining on in-domain data. |
| Approach: | They propose a pretrained sequence-to-sequence language model that can handle different parts of dialogue belonging to multiple speakers and combine them to produce a coherent monologue summary. |
| Outcome: | The proposed techniques outperform baseline models on a dialogue summarization dataset. |
Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020.emnlp-main)
Copied to clipboard
Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan
| Challenge: | Existing models for temporal ordering of events rely on pretrained representations, transfer and multitask learning, and self-training techniques. |
| Approach: | They propose a neural architecture and a set of training methods for ordering events by predicting temporal relations by pre-training models. |
| Outcome: | The proposed models can predict temporal relations between two pairs of events within a span of text and identify temporal relationships between them. |
Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning (P19-1)
Copied to clipboard
| Challenge: | Abstract meaning representations (AMRs) are labeled directed acyclic graphs that represent a non intersentential abstraction of natural language with broad-coverage semantic representations. |
| Approach: | They build upon a transition-based AMR parser that uses Stack-LSTMs and augment training with policy learning. |
| Outcome: | The proposed parser performs comparable to the best published parsers. |
On the evolution of syntactic information encoded by BERT’s contextualized representations (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on how linguistic information is encoded in pretrained language models to solve supervised tasks. |
| Approach: | They analyze how the syntax trees are embedded in the geometry of pretrained models for six different tasks, covering all levels of the linguistic structure. |
| Outcome: | The proposed model is able to learn and improve on GLUE and SQUAD, but it lacks the ability to learn the linguistic information required to solve the tasks. |
Label Semantics for Few Shot Named Entity Recognition (2022.findings-acl)
Copied to clipboard
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, Dan Roth
| Challenge: | Named entity recognition (NER) is a fundamental natural language understanding task that requires large amounts of high quality annotated in-domain data. |
| Approach: | They propose a neural architecture that leverages the semantic information in the names of the labels to give the model additional signal and enriched priors. |
| Outcome: | The proposed model is especially effective in low resource settings. |
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not investigated the relationship between a token's frequency in the training corpus and syntactic properties models learn about it. |
| Approach: | They develop controlled experiments that probe models’ syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. |
| Outcome: | The proposed models can make syntactic generalizations for tokens seen as few as two times during training and transfer them to transformed contexts. |
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent smaller language models rely on synthetic data generated using larger Language models. |
| Approach: | They propose a method for generating synthetic data that enhances diversity through meta-prompting . they use 25 million tokens of synthetic data generated by a language model orchestrated by multiple “expert” LLM agents to collaboratively generate data. |
| Outcome: | The proposed method outperforms the base LLM in Finance and Biomedicine with 25 million tokens of synthetic data. |
A Weak Supervision Approach for Few-Shot Aspect Based Sentiment Analysis (2024.eacl-long)
Copied to clipboard
Robert Vacareanu, Siddharth Varia, Kishaloy Halder, Shuai Wang, Giovanni Paolini, Neha Anna John, Miguel Ballesteros, Smaranda Muresan
| Challenge: | Existing methods to improve few-shot performance in aspect-based sentiment analysis (ABSA) require complex interactions between the target and the polarity of the sentiment. |
| Approach: | They propose a pipeline approach to construct a noisy ABSA dataset and adapt it to the ABSA tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art on the aspect extraction sentiment classification task and is capable of performing the harder aspect sentiment triplet extraction task. |
Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2020 focus on semi-structured document classification in a zero-shot setting . positional, layout, and style information play a vital role in interpreting such documents . |
| Approach: | They propose a matching-based approach that relies on a pairwise contrastive objective for pretraining and fine-tuning. |
| Outcome: | The proposed method significantly improves Macro F1 in the zero-shot learning setting. |
Linking Entities to Unseen Knowledge Bases with Arbitrary Schemas (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on entity linking relies on a knowledge base that is not known at training time. |
| Approach: | They propose a method to flexibly convert entities with several attribute-value pairs from arbitrary KBs into flat strings and use it to generalize the model. |
| Outcome: | The proposed model is 12% more accurate than baseline models on English datasets. |
Event-Driven News Stream Clustering using Entity-Aware Contextual Embeddings (2021.eacl-main)
Copied to clipboard
| Challenge: | a novel method for online news stream clustering is proposed . a user can scour the many news sources multiple times a day to find news articles . |
| Approach: | They propose a method for online news stream clustering that is a variant of the streaming K-means algorithm. |
| Outcome: | The proposed model achieves state-of-the-art on a standard stream clustering dataset of English documents. |
Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data (2026.findings-acl)
Copied to clipboard
Xuanming Zhang, Shwan Ashrafi, Aziza Mirsaidova, Amir H. Rezaeian, Miguel Ballesteros, Lydia Chilton, Zhou Yu, Dan Roth
| Challenge: | Recent work has explored reasoning efficiency via test-time scaling and early exit strategies. |
| Approach: | They propose an anytime reasoning framework and the Anytime Index to improve model quality . they also propose an inference-time self-improvement method to produce better intermediate solutions . |
| Outcome: | The proposed method improves on NaturalPlan, AIME, and GPQA datasets and improves reasoning quality and efficiency under budget constraints. |
Comparing Biases and the Impact of Multilingual Training across Multiple Languages (2023.emnlp-main)
Copied to clipboard
Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, Dan Roth
| Challenge: | Currently, studies on bias and fairness in natural language processing focus on a single language and/or across few attributes (e.g. gender, race). However, biases can manifest differently across languages for individual attributes. |
| Approach: | They adapt existing sentiment bias templates in English to Italian, Chinese, Hebrew, and Spanish for race, religion, nationality, and gender. |
| Outcome: | The proposed model favors groups that are dominant in each language's culture, indicating bias amplification, after multilingual finetuning. |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (2023.eacl-main)
Copied to clipboard
| Challenge: | Temporal concept drift is a problem of data changing over time. |
| Approach: | They benchmark 11 pretrained masked language models on a series of tests to evaluate temporal concept drift. |
| Outcome: | The proposed framework evaluates 11 pretrained masked language models on a series of tests . it aims to reveal how robust an MLM is over time and provide a signal in case it has become outdated . |
Recursive Subtree Composition in LSTM-Based Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing studies show that tree structure modelling on top of sequence modelling is not feasible. |
| Approach: | They propose to recursively compose subtree representations in a biLSTM-based parser to capture subtreas. |
| Outcome: | The proposed model improves performance under ablating the backward LSTM and the forward LS. |
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
| Challenge: | LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone. |
| Approach: | They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs. |
| Outcome: | The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs. |
Taxonomy Expansion for Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
Karthikeyan K, Yogarshi Vyas, Jie Ma, Giovanni Paolini, Neha John, Shuai Wang, Yassine Benajiba, Vittorio Castelli, Dan Roth, Miguel Ballesteros
| Challenge: | Training a Named Entity Recognition model involves fixing a taxonomy of entity types . however, requirements evolve and a model may need to recognize additional entity types. |
| Approach: | They propose a method that uses only partially annotated datasets to train a model to recognize additional entity types. |
| Outcome: | The proposed approach performs better with partially annotated datasets than other approaches . the gap between the proposed approach and other approaches is large in additional datasets . |
Resource-Enhanced Neural Model for Event Argument Extraction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on event argument extraction (EE) is limited due to data scarcity and lack of a model encoder. |
| Approach: | They propose to capture the long-range dependency between an event trigger and a distant event argument using unlabeled data. |
| Outcome: | Experiments on the English ACE 2005 benchmark show that the proposed method achieves a new state-of-the-art. |
Transition-based Parsing with Stack-Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing parsing systems use local or global models of the parser state to improve performance. |
| Approach: | They propose to modify the sequence-to-sequence Transformer to model global or local parser states in transition-based parsing. |
| Outcome: | The proposed model significantly improves performance on dependency and Abstract Meaning Representation (AMR) parsing tasks. |
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (2026.findings-acl)
Copied to clipboard
Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi, Dan Roth
| Challenge: | Large language models suffer performance degradation when user instructions and context are distributed over multiple conversational turns. |
| Approach: | They propose a framework that condenses chat history in the background without disrupting the user experience. |
| Outcome: | The proposed framework reduces token counts by up to 72% in 10-turn dialogues while remaining robust to distractors and irrelevant turns. |
Multilingual Neural Machine Translation with Task-Specific Attention (C18-1)
Copied to clipboard
| Challenge: | Multilingual machine translation is a task of building a system capable of translating between multiple source and target languages. |
| Approach: | They propose task-specific attention models to retain parameter sharing generalization . they observe improved translation quality even in low-resource zero-shot directions . |
| Outcome: | The proposed model retains parameter sharing generalization while allowing language-specific specialization . it improves translation quality even in low-resource zero-shot translation directions . |
Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | generating aspect-specific and general opinion summaries is challenging due to the lack of annotated data. |
| Approach: | They propose two unsupervised approaches to generate aspect-specific and general opinion summaries by training on synthetic datasets constructed with aspect-related review contents. |
| Outcome: | The proposed method outperforms existing methods on space and Oposum+ and on other metrics. |
To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging (2020.emnlp-main)
Copied to clipboard
Kasturi Bhattacharjee, Miguel Ballesteros, Rishita Anubhai, Smaranda Muresan, Jie Ma, Faisal Ladhak, Yaser Al-Onaizan
| Challenge: | Using large amounts of unlabeled data to improve performance has become the foundation for many natural language processing tasks. |
| Approach: | They propose a task-specific semi-supervised approach that uses unlabeled data in a more task-agnostic manner. |
| Outcome: | The proposed approach achieves similar performance to BERT on a set of sequence tagging tasks with less financial and environmental impact. |
Pieces of Eight: 8-bit Neural Machine Translation (N18-3)
Copied to clipboard
| Challenge: | Neural machine translation models are trained using 32-bit floating point values and have improved fluency and adequacy. |
| Approach: | They propose to use 8-bit quantization to train models using 32-bit floating point values and show that 8- bit translation makes a non-negligible impact in terms of speed with no degradation in accuracy and adequacy. |
| Outcome: | The proposed method improves accuracy and accuracy without degradation in accuracy and adequacy. |
JTPRO: A Joint Tool–Prompt Reflective Optimization Framework for Language Agents (2026.findings-acl)
Copied to clipboard
Sandip Ghoshal, Anshul Mittal, Jyotika Singh, Miguel Ballesteros, Weiyi Sun, Fang Tu, Shailender Singh, Yassine Benajiba, Fahad Shah, Sujeeth Bharadwaj, Sujith Ravi, Dan Roth
| Challenge: | Large language model agents struggle with ambiguous tool descriptions and underspecified tool schemas that ignore tool-specific nuances. |
| Approach: | They propose a framework for improving tool-calling reliability in trace-supervised settings by rolling out-driven reflection. |
| Outcome: | The proposed framework outperforms baselines and reflective prompt optimizers by 5%–20% on OSR. |
Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidth (2026.eacl-long)
Copied to clipboard
Michelle Yuan, Weiyi Sun, Amir H. Rezaeian, Jyotika Singh, Sandip Ghoshal, Yao-Ting Wang, Miguel Ballesteros, Yassine Benajiba
| Challenge: | despite advances in transformers, their theoretical limitations in discrete reasoning remain a critical open problem. |
| Approach: | They synthesize recent advances from three theoretical perspectives to clarify structural and computational barriers transformers face when performing symbolic computations. |
| Outcome: | The proposed models excel at pattern matching and interpolation, but they face bottlenecks in communication and depth constraints. |
Using Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Large pretrained language models can generate powerful text but cannot be controlled at a sub-sentential level. |
| Approach: | They propose to make such fine-grained control possible in pretrained LMs by generating text directly from a semantic representation, Abstract Meaning Representation (BART), which is augmented at the node level with syntactic control tags. |
| Outcome: | The proposed method can generate text from a semantic representation, which is augmented at the node level with syntactic control tags. |