Papers by Ellie Pavlick
A Visuospatial Dataset for Naturalistic Verb Learning (2020.starsem-1)
Copied to clipboard
| Challenge: | a new dataset is available for training and evaluating grounded language models . our data is designed to emulate the quality of language data a pre-verbal child would have access to . |
| Approach: | They propose a dataset for training and evaluating grounded language models . they use naturalistic, spontaneous speech paired with richly grounded visuospatial context . |
| Outcome: | The proposed dataset compares two distributional semantics models with one that does not. |
WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse (D18-1)
Copied to clipboard
| Challenge: | a corpus of 43 million atomic edits is available for Wikipedia edit history . edits are instances in which a human editor has inserted a single contiguous phrase into, or deleted a contigous phrase from, an existing sentence. |
| Approach: | They use Wikipedia edit history to mine atomic edits across 8 languages . they find edits contain instances in which a human editor has inserted a single phrase into, or deleted a contiguous phrase from, an existing sentence. |
| Outcome: | The data show that edits differ from the language observed in standard corpora and that models trained on edits encode different aspects of semantics and discourse than models trained in raw text. |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (D18-1)
Copied to clipboard
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme
| Challenge: | a plethora of new natural language inference datasets has been created in recent years . however, these datasets do not provide clear insight into what type of reasoning or inference a model may be performing. |
| Approach: | They propose to recast 13 existing natural language inference datasets into a common structure. |
| Outcome: | The proposed datasets provide insight into how well a sentence representation captures distinct types of reasoning. |
Unit Testing for Concepts in Neural Networks (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing theories of language and cognition hold that these representations are structured in a compositional way and that the meanings of composite concepts (''gray car'') are inherited predictably from the meaning of the parts. |
| Approach: | They propose to test models for determining whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. |
| Outcome: | The proposed models succeed on tests of groundedness, modularity, and reusability of concepts, but important questions about causality remain open. |
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models (LLMs) exhibit factual inconsistencies across languages . authors identify two primary sources of error: insufficient engagement of reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer. |
| Approach: | They propose two vector interventions to redirect the model toward better internal paths for higher factual consistency. |
| Outcome: | The proposed interventions increase the recall accuracy by over 35 percent for the lowest-performing language. |
Are Language Models Worse than Humans at Following Prompts? It’s Complicated (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models perform surprisingly well when given intentionally irrelevant or misleading prompts. |
| Approach: | They challenge an assumption that humans would perform badly when given pathological instructions by ignoring irrelevant prompts and following them faithfully when given misleading instructions. |
| Outcome: | The proposed model performs well when given intentionally irrelevant or misleading prompts, whereas models do not. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
How well do NLI models capture verb veridicality? (D19-1)
Copied to clipboard
| Challenge: | In natural language inference, contexts are considered veridical if they allow us to infer that their underlying propositions make true claims about the real world. |
| Approach: | They propose to use a dataset for veridicality evaluation consisting of 1,500 sentence pairs, covering 137 unique verbs. |
| Outcome: | The proposed model learns to make correct inferences about veridicality in verb-complement constructions. |
Interpretability and Analysis in Neural NLP (2020.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to introduce the nascent field of interpretability and analysis of neural networks in NLP . |
| Approach: | This tutorial will introduce the nascent field of interpretability and analysis of neural networks in NLP. |
| Outcome: | This tutorial will introduce the nascent field of interpretability and analysis of neural networks in NLP. |
Analyzing Modular Approaches for Visual Question Decomposition (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modular neural networks without additional training have been shown to surpass end-to-end neural networks on challenging vision–language tasks. |
| Approach: | They propose to use BLIP-2-based modular neural networks without additional training to build programs and a number of skill-specific, task-oriented modules to execute them. |
| Outcome: | The proposed methods outperform end-to-end neural networks on vision language tasks and retain performance when they use task-agnostic selections. |
Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models encode commonsense relational knowledge, but they are often insensitive to context, ignoring overt contextual cues such as negations. |
| Approach: | They propose a procedure that exploits generic associations in masked language models to create model-specific entailment schemas. |
| Outcome: | The proposed procedure exploits generic associations in masked language models to create model-specific Winograd-style entailment schemas. |
“Was it “stated” or was it “claimed”?: How linguistic bias affects generative language models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Several studies have identified such linguistic classes of words that occur frequently in natural language text and are bias-inducing by virtue of their framing effects. |
| Approach: | They propose to use linguistic cues to induce subtle biases through implied sentiment and presupposed facts to influence the distribution of the generated text. |
| Outcome: | The proposed models are sensitive to these framing effects, but show that they lead to measurable style and topic differences in the generated text, leading to language that is, on average, more polarised and more skewed towards controversial entities and events. |
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing Question-Answering (QA) datasets contain unanswerable questions . however, their treatment in QA systems remains primitive . |
| Approach: | They propose a framework that provides answers based on presupposition failure over oracle behavior of existing QA systems. |
| Outcome: | The proposed system provides responses based on presupposition failure over oracle behavior of existing QA systems. |
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. |
| Approach: | They propose a synthetic task, Multilingual Othello, as a testbed to investigate the factors that contribute to the learning of a language-neutral representation. |
| Outcome: | The proposed approach induces the learning of language-neutral representation and facilitates cross-lingual transfer. |
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. |
| Approach: | They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts . |
| Outcome: | The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts. |
Does Vision-and-Language Pretraining Improve Lexical Grounding? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pretrained language models (LMs) have been criticized for lack of grounding, i.e., connecting words to their meanings in the physical world. |
| Approach: | They compare vision-and-language (VL) models trained jointly on text and image or video data to find out how they compare to text-only counterparts. |
| Outcome: | The proposed model outperforms the text-only variants on a commonsense question answering task. |
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | lexical overlap heuristics are effective for frequent example types but break down in more challenging cases. |
| Approach: | They propose to use a set of examples to test whether a sentence entails another . they propose to adopt three fallible syntactic heuristics for statistical NLI models . |
| Outcome: | The proposed model performs poorly on the HANS dataset, suggesting they have adopted these heuristics. |
Do Trajectories Encode Verb Meaning? (2022.naacl-main)
Copied to clipboard
| Challenge: | Distributional models learn representations of words from text but lack grounding or the linking of text to the non-linguistic world. |
| Approach: | They investigate the extent to which trajectories naturally encode verb semantics . they build a procedurally generated agent-object-interaction dataset and compare methods . |
| Outcome: | The proposed model can capture verb semantics by tracing trajectories and self-supervised pretraining. |
Frequency Effects on Syntactic Rule Learning in Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. |
| Approach: | They investigate the performance of BERT on English subject–verb agreement by analyzing word frequency and absolute frequency of verb forms. |
| Outcome: | The proposed model generalizes well to subject–verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. |
Emergence of Abstract State Representations in Embodied Sequence Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sequence modeling is a simple yet versatile task that can be applied to more complex decision-making domains. |
| Approach: | They build a sequence modeling Transformer which takes a language instruction, actions, and environmental observations as inputs and then trains a model to reconstruct environmental layouts. |
| Outcome: | The proposed model can reconstruct environmental layouts from the inputs of the model and language instructions play a role in the reconstruction accuracy. |
BERT Rediscovers the Classical NLP Pipeline (P19-1)
Copied to clipboard
| Challenge: | Pre-trained text encoders have advanced the state of the art on many NLP tasks . Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically . |
| Approach: | They aim to quantify where linguistic information is captured within a network model . they aim to use pre-trained text encoders to displace static word embeddings . |
| Outcome: | The proposed model can adjust the pipeline dynamically, revealing lower-level decisions on the basis of disambiguation from higher-level representations. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Learning Scalar Adjective Intensity from Paraphrases (D18-1)
Copied to clipboard
| Challenge: | Existing lexical resources do not include the relative intensities of adjectives. |
| Approach: | They propose a method to automatically learn relative intensity relation between scalar adjectives . they use a paraphrase-based method that assumes that a pair of adjectives is "really hot" a similar method is used to infer the polarity of indirect answers to "yes/no" questions . |
| Outcome: | The proposed method improves the quality of systems for ordering sets of scalar adjectives and inferring the polarity of indirect answers to "yes/no" questions. |
A Knapsack by Any Other Name: Presentation impacts LLM performance on NP-hard problems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EHOP is a collection of NP-hard optimization problems expressed in natural language . state-of-the-art LLMs solve textbook problems more accurately than their real-life counterparts, but they lack a truly robust reasoning mechanism. |
| Approach: | They introduce a dataset of everyday hard optimization problems (EHOP) which includes problem formulations found in computer science textbooks, versions dressed up as problems that could arise in real life, and variants with inverted rules. |
| Outcome: | The proposed dataset includes problem formulations found in computer science textbooks, versions dressed up as problems that could arise in real life, and variants with inverted rules. |
Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces (2020.emnlp-main)
Copied to clipboard
| Challenge: | popular pretrained models encode both denotation and connotation as one entangled representation . a researcher using a pretrained representation can confuse words with connotations . |
| Approach: | They propose a nerual netowrk that decomposes a pretrained representation as independent denotation and connotation representations. |
| Outcome: | The proposed model improves document rankings by comparing denotation and connotation representations with extrinsic representations. |
AND does not mean OR: Using Formal Languages to Study Language Models’ Representations (2021.acl-short)
Copied to clipboard
| Challenge: | A current open question in natural language processing is to what extent language models are able to capture the meaning of language. |
| Approach: | They propose to simulate a distributional language model’s ability to differentiate logical symbols using motivated constraints and motivated constraints. |
| Outcome: | The results show that the proposed models are unable to differentiate meaningfully different symbols, suggesting a limitation to the types of semantic signals that current models are capable of exploiting. |
Pretraining on Interactions for Learning Grounded Affordance Representations (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing studies of affordances have not integrated into formal semantics. |
| Approach: | They propose to integrate 3D objects' trajectories into a neural network to predict their traversories. |
| Outcome: | The proposed model outperforms 2D computer vision models and is more accurate than expected. |
Does CLIP Bind Concepts? Probing Compositionality in Large Image Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large-scale neural network models combining text and images have made incredible progress in recent years, but to what extent they encode compositional representations of the concepts over which they operate remains an open question . |
| Approach: | They compare the performance of a large pretrained vision and language model (CLIP) to a set of three synthetic datasets designed to test concept binding. |
| Outcome: | The proposed model can encode compositional concepts and bind variables in a structure-sensitive way, e.g., differentiating ‘cube behind sphere’ from ‘cub behind cube’. |
Language Models Implement Simple Word2Vec-style Vector Arithmetic (2024.naacl-long)
Copied to clipboard
| Challenge: | a primary criticism of language models is their inscrutability. |
| Approach: | They propose to use a vector arithmetic style mechanism to solve relational tasks . they find that this mechanism is specific to tasks that require retrieval from pretraining memory . |
| Outcome: | The proposed model reduces to a simple additive update for a variety of tasks . the findings contribute to proving that the models are interpretable and reliable . |