Papers by Naomi Saphra

Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)

Copied to clipboard

Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew Leavitt

Challenge:	Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate.
Approach:	They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively.
Outcome:	The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines.

Using Shapley interactions to understand how models use structure (2025.acl-long)

Copied to clipboard

Divyansh Singhvi, Diganta Misra, Andrej Erkelens, Raghav Jain, Isabel Papadimitriou, Naomi Saphra

Syntactic Text Processing

Challenge:	Language models are intricately structured systems, and attribution measures are important for understanding how they combine features to influence outputs.
Approach:	They use Shapley Taylor interaction indices to examine how language and speech models internally relate and structure their inputs.
Outcome:	The proposed methods show that language models encode phonetic interactions . they show that the inputs are more entangled for pairs where a consonant influences a vowel or approximant .

Understanding Learning Dynamics Of Language Models with SVCCA (N19-1)

Copied to clipboard

Naomi Saphra, Adam Lopez

Language Models

Challenge:	a new study shows that neural models implicitly encode linguistic features . but no research shows how these encodings arise as the models are trained .
Approach:	They propose a method that compares learning across time and across models using annotated data.
Outcome:	The proposed method compares learned representations across time and across models without evaluation on annotated data.

Fast Forwarding Low-Rank Training (2024.emnlp-main)

Copied to clipboard

Adir Rahamim, Naomi Saphra, Sara Kangaslahti, Yonatan Belinkov

Green & Sustainable NLP

Challenge:	Modern optimizers provide a spectacular array of tweaks to stabilize training trajectories and accelerate Stochastic Gradient Descent (SGD).
Approach:	They propose a fast-forward approach to accelerate large segments of SGD training . they alternate between Adam SGD for burn-in and accelerating by line search .
Outcome:	The proposed approach speeds up training without compromising model performance.

First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models (2024.naacl-long)

Copied to clipboard

Naomi Saphra, Eve Fleisig, Kyunghyun Cho, Adam Lopez

Language Models

Challenge:	a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible.
Approach:	They argue that disparities in scale are transient and researchers can work to reduce them . they argue that data, rather than hardware, is still a bottleneck for many applications .
Outcome:	a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible.

Pareto Probing: Trading Off Accuracy for Complexity (2020.emnlp-main)

Copied to clipboard

Tiago Pimentel, Naomi Saphra, Adina Williams, Ryan Cotterell

Representation Learning

Challenge:	Neural networks are a pillar of modern NLP systems, but their inner workings are poorly understood.
Approach:	They propose a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume.
Outcome:	The proposed probe metric conforms to accepted rankings among contextual representations, and is more complex than other probe tasks.

ChatGPT Doesn’t Trust Chargers Fans: Guardrail Sensitivity in Context (2024.emnlp-main)

Copied to clipboard

Victoria Li, Yida Chen, Naomi Saphra

Language Models

Challenge:	Existing work addresses the limitations of chatbot guardrails, which limit responses to uncertain or sensitive questions.
Approach:	They generate user biographies that offer ideological and demographic information about the user.
Outcome:	The proposed model can infer a likely political ideology and modify guardrail behavior accordingly.

LSTMs Compose—and Learn—Bottom-Up (2020.findings-emnlp)

Copied to clipboard

Naomi Saphra, Adam Lopez

Language Models

Challenge:	Recent work in NLP shows that LSTMs capture compositional structure in language data.
Approach:	They propose to measure the decompositional interdependence between word meanings in an LSTM based on their gate interactions.
Outcome:	The proposed model can model syntactic relationships rather than learning the longer-range relations independently.

Benchmarking Compositionality with Formal Languages (2022.coling-1)

Copied to clipboard

Josef Valvoda, Naomi Saphra, Jonathan Rawski, Adina Williams, Ryan Cotterell

Challenge:	Compositionality is a hallmark of human language, but it is not yet fully understood . recombining known primitive concepts into larger novel combinations is elusive .
Approach:	They use finite-state transducers to make a dataset with controllable compositionality . they find that the models either learn the relations completely or not at all .
Outcome:	The proposed model learns the relation completely or not at all on large datasets.

Attribute Diversity Determines the Systematicity Gap in VQA (2024.emnlp-main)

Copied to clipboard

Ian Berlot-Attwell, Kumar Agrawal, Annabelle Carrell, Yash Sharma, Naomi Saphra

Challenge:	a systematicity gap exists between neural networks generalizing to new combinations of familiar concepts . conventionally trained neural networks struggle to generalize systematically .
Approach:	They propose to train a visual question answering model with CLEVR-HOPE as a diagnostic dataset to test this hypothesis.
Outcome:	The systematicity gap is reduced by increasing the diversity of training data, the authors show . the authors suggest that the more distinct attribute type combinations are seen during training, the more systematic the model will be.

Data Drives Unstable Hierarchical Generalization in LMs (2025.emnlp-main)

Copied to clipboard

Tian Qin, Naomi Saphra, David Alvarez-Melis

Syntactic Text Processing

Challenge:	Early in training, LMs can behave like n-gram models but eventually learn tree-based syntactic rules and generalize out of distribution (OOD).
Approach:	They study how complex data drives hierarchical rules, while less complex encourages shortcut learning . they find a model uses rules to generalize if its training data is diverse .
Outcome:	The proposed model learns to generalize hierarchically if its training data is complex . a model learn if it includes center-embedded clauses, a special syntactic structure .

A Non-Linear Structural Probe (2021.naacl-main)

Copied to clipboard

Jennifer C. White, Tiago Pimentel, Naomi Saphra, Ryan Cotterell

Syntactic Text Processing

Challenge:	Probing is a method of investigating the encoding of knowledge in contextual representations.
Approach:	They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe.
Outcome:	The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages.