Papers by Roy Schwartz
ABC: Attention with Bounded-memory Control (2022.acl-long)
Copied to clipboard
Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, Noah A. Smith
| Challenge: | Existing approaches to attention with bounded-memory control (ABC) have a quadratic complexity in sequence lengths, making it prohibitive for long sequences. |
| Approach: | They propose a new abstraction that bounds memory size to improve efficiency . they propose bounded-memory control, which connects several efficient attention variants . |
| Outcome: | The proposed approach outperforms existing approaches on language modeling, machine translation, and masked language model finetuning. |
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
Knowledge Enhanced Contextual Word Representations (D19-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
| Challenge: | Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge. |
| Approach: | They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism. |
| Outcome: | The proposed model improves perplexity, ability to recall facts and word sense disambiguation. |
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models often encode word-form variation as linear directions in the embedding space. |
| Approach: | They propose a compact reshaping of large language models' vocabulary by using shared vectors instead of unique tokens. |
| Outcome: | The proposed approach frees 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. |
A Formal Hierarchy of RNN Architectures (2020.acl-main)
Copied to clipboard
| Challenge: | Existing theories of expressive power of RNNs are limited. |
| Approach: | They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence. |
| Outcome: | The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models. |
On Pruning State-Space LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
Curating Datasets for Better Performance with Example Training Dynamics (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve data quality but rely on data quantity to improve performance are not effective. |
| Approach: | They propose a method for weighing the relative importance of examples in a dataset based on their Example Training dynamics (ETD) they propose an active learning approach for computing ETD during training rather than as a preprocessing step. |
| Outcome: | The proposed method can be used to improve performance in in-distribution and out-of-distortion testing. |
A Mixture of h - 1 Heads is Better than h Heads (2020.acl-main)
Copied to clipboard
| Challenge: | Evidence has shown that multi-head attentive neural architectures are overparameterized. |
| Approach: | They propose a multi-head attentive neural architecture that “reallocates” attention heads to different inputs. |
| Outcome: | The proposed model outperforms baselines on machine translation and language modeling tasks. |
Fighting Bias With Bias: Promoting Model Robustness by Amplifying Dataset Biases (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work sought to develop robust, unbiased models by filtering biased examples from training sets. |
| Approach: | They propose to filter out biased examples from training sets to improve models' performance. |
| Outcome: | The proposed evaluation framework is more challenging than the original dataset splits and even more challenging that hand-crafted challenge sets. |
The Right Tool for the Job: Matching Model and Instance Complexities (2020.acl-main)
Copied to clipboard
| Challenge: | a large increase in the size of NLP models can increase production costs and reduce adoption on real-time devices. |
| Approach: | They propose a modification to contextual representation fine-tuning which allows for an early exit from neural network calculations for simple instances and late exit for hard instances. |
| Outcome: | The proposed method produces models which are up to five times faster than the state of the art while preserving their accuracy. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (D18-1)
Copied to clipboard
| Challenge: | a new dataset presents a task of grounded commonsense inference, unifying natural language inference and commonsensical reasoning. |
| Approach: | They propose a procedure that constructs a de-biased dataset by iteratively training stylistic classifiers and using them to filter the data. |
| Outcome: | The proposed procedure oversamples a de-biased dataset using state-of-the-art language models . human models struggle on the proposed procedure, indicating significant opportunities for future research. |
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evidence is emerging that neural networks learn due to inductive bias in the training routine, typically a variant of gradient descent (GD). |
| Approach: | They propose to characterize GD as an inductive bias in transformer training . they document norm growth in transformer language models and show they are saturated . |
| Outcome: | Empirically, we document norm growth in the training of transformer language models . the results suggest saturation is a new characterization of an inductive bias implicit in GD . |
Bridging CNNs, RNNs, and Weighted Finite-State Machines (P18-1)
Copied to clipboard
| Challenge: | recurrent and convolutional neural networks are useful for encoding natural language utterances. |
| Approach: | They propose a model that combines neural representation learning with weighted finite-state automatas to learn a soft version of traditional surface patterns. |
| Outcome: | The proposed model is comparable or better than a BiLSTM baseline and a CNN baseline on three text classification tasks. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Expected Validation Performance and Estimation of a Random Variable’s Maximum (2021.findings-emnlp)
Copied to clipboard
| Challenge: | In this paper, we analyze three statistical estimators for expected validation performance . Often researchers only report the performance of the best-found model during a hyperparameter search . |
| Approach: | They analyze three estimators for expected validation performance to compare models . they find that the estimator with the smallest variance has the largest bias . |
| Outcome: | The proposed model has the highest variance and the estimator with the smallest variance has the largest bias. |
RNN Architecture Learning with Sparse Regularization (D19-1)
Copied to clipboard
| Challenge: | State-of-the-art NLP models require hundreds of millions and even billions of parameters to perform, which can lead to memory usage and increased runtime. |
| Approach: | They propose a structure learning method that uses group lasso to learn sparse, parameter-efficient NLP models by pruning more than 90% of the weights of rational RNNs. |
| Outcome: | The proposed method learns sparse, parameter-efficient models without sacrificing performance relative to parameter-rich baselines. |
Data Contamination: From Memorization to Exploitation (2022.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models are typically trained on web-based datasets that are often "contaminated" with downstream test sets. |
| Approach: | They propose a method to pretrain BERT models on Wikipedia and labeled downstream datasets and fine-tune them on the relevant task. |
| Outcome: | The proposed method compares models on Wikipedia and labeled downstream datasets on two models and three downstream tasks. |
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that supervised models exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. |
| Approach: | They propose a method which automatically generates contrast sets for the visual question answering task by using a semantic input representation. |
| Outcome: | The proposed method computes the answer of perturbed questions, thus reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects. |
Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer-based language models such as BERT, DeBERTa, and GPT3 have become the go-to tool in NLP. |
| Approach: | They propose an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. |
| Outcome: | The proposed method outperforms Early-Exit and Multi-Model at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. |
Show Your Work: Improved Reporting of Experimental Results (D19-1)
Copied to clipboard
| Challenge: | Current practice is to train multiple instantiations of each, choose the best model of each type, and compare their performance on held-out test data. |
| Approach: | They propose to measure expected validation accuracy as a function of computation budget . authors find comparisons where authors would have reached different conclusions if they had used more computation . |
| Outcome: | The proposed method shows that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. |
How Quantization Shapes Bias in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | a systematic review of quantization's effects on model biases focuses on stereotypes, fairness, toxicity, and sentiment. |
| Approach: | They focus on weight and activation quantization strategies and examine their effects across bias types including stereotypes, fairness, toxicity, and sentiment. |
| Outcome: | The proposed method can reduce stereotypes and unfairness, but it tends to increase stereotypes in generative tasks. |
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications (N18-1)
Copied to clipboard
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
| Challenge: | a dataset of 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR is presented to study peer reviews. |
| Approach: | They propose to use the dataset to collect peer reviews from top-tier venues including ACL, NIPS and ICLR and to use it to create a dataset of peer reviews for research purposes. |
| Outcome: | The proposed dataset includes 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. |
Transformers are Multi-State RNNs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Schwartz et al., 2017) have been using transformers for long-range tasks for NLP since the 1990s. |
| Approach: | They propose a transformer-only transformer with unlimited hidden state size that can be converted into bounded multistate RNNs by fixing the size of their hidden state. |
| Outcome: | The proposed compression policy outperforms baseline compression policies on long range tasks and LLMs. |
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval (2025.findings-emnlp)
Copied to clipboard
Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng
| Challenge: | Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. |
| Approach: | They propose a model-agnostic mitigation strategy that transforms a long-context task into a short-concept one by prompting the model to recite the retrieved evidence before attempting to solve the problem. |
| Outcome: | The proposed model improves on a long-context task up to 4% on RULER. |
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work shows that deep learning models are sensitive to low-level correlations between simple features and specific output labels, leading to over-fitting and lack of generalization. |
| Approach: | They propose to eliminate single-word correlations altogether to mitigate this problem . they highlight several alternatives to dataset balancing to enhance contexts . |
| Outcome: | The proposed approach to balancing datasets is insufficient, the authors argue . they suggest enhancing datasets with richer contexts and abstaining from interaction . |
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020.emnlp-main)
Copied to clipboard
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, Yejin Choi
| Challenge: | Large datasets have become commonplace in NLP research, but the emphasis on quantity has made it challenging to assess the quality of data. |
| Approach: | They propose a model-based tool to characterize and diagnose large datasets . they leverage the behavior of the model on individual instances during training . |
| Outcome: | Experiments on four datasets show that the tool can characterize and diagnose datasets with a model-based tool. |
PaLM: A Hybrid Parser and Language Model (D19-1)
Copied to clipboard
| Challenge: | Recent language models have shown strong data-fitting performance, but do not explicitly encode any notion of structural information. |
| Approach: | They propose a hybrid parser and neural language model that adds an attention layer over text spans in the left context. |
| Outcome: | The proposed model outperforms baseline models on language modeling and provides syntactically-informed representations of the context. |
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. |
| Approach: | They investigate how semantic information is distributed across token representations in text-to-image prompts by patching techniques to uncover encoding patterns. |
| Outcome: | The proposed model can improve alignment and generation quality by modifying the diffusion stage and the cross-attention mechanism. |
Extracting a Knowledge Base of Mechanisms from COVID-19 Papers (2021.naacl-main)
Copied to clipboard
Tom Hope, Aida Amini, David Wadden, Madeleine van Zuylen, Sravanthi Parasa, Eric Horvitz, Daniel Weld, Roy Schwartz, Hannaneh Hajishirzi
| Challenge: | COVID-19 has spawned a diverse body of scientific literature that is challenging to navigate . researchers are using automated tools to help find useful knowledge . |
| Approach: | They develop a schema to extract mechanism relations from scientific papers . their search engine, dataset and code are publicly available . |
| Outcome: | The proposed schema outperforms PubMed search in clinical trials. |
Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand? (2021.tacl-1)
Copied to clipboard
| Challenge: | Language models trained on billions of tokens have recently led to unprecedented results on many NLP tasks. |
| Approach: | They examine whether assertions enable a system to emulate representations preserving semantic relations like equivalence. |
| Outcome: | The proposed model can emulate representations preserving semantic relations like equivalence, but it can become uncomputable for classes of languages where expressions can take different values in different contexts. |
Rational Recurrences (D18-1)
Copied to clipboard
| Challenge: | Recent studies show that neural models lack strong intuitions . recent studies show connections between convolutional neural networks and weighted finite state automata (WFSAs) |
| Approach: | They show that some recurrent neural networks share a connection to weighted finite state automata (WFSAs) they define rational recurrences as recursive hidden state update functions . they propose to use these functions to write forward calculations of a finite set of WFSA's . |
| Outcome: | The proposed model outperforms two baselines on language modeling and text classification. |
Data Efficient Masked Language Modeling for Vision and Language (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. |
| Approach: | They propose a masking strategy that masks tokens with a 15% probability for text-only data. |
| Outcome: | The proposed masking strategy outperforms the baseline model on a prompt-based probing task designed to elicit image objects. |
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets (N19-1)
Copied to clipboard
| Challenge: | Several datasets have been constructed to expose brittleness in models trained on existing benchmarks. |
| Approach: | They propose to use a challenge dataset to examine model adaptations by exposing models to a metaphorical pathogen and assessing how well they can adapt. |
| Outcome: | The proposed method analyzes the NLI stress tests and the Adversarial SQuAD datasets and shows that they are no longer challenging and others remain difficult. |