Papers by Allyson Ettinger
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
On the Interplay Between Fine-tuning and Composition in Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer language models have shown remarkable performance on a variety of NLP tasks. |
| Approach: | They propose to fine-tune transformer language models on a paraphrase and sentiment task and analyze their results to determine whether they benefit compositionality. |
| Outcome: | The proposed model performance on a paraphrase and sentiment task is compared with pre-trained models on lexical-level representations. |
“You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate proficiency and fluency in the use of language, but do they have the linguistic knowledge to serve as an expert linguistic annotator? |
| Approach: | They examine the successes and limitations of large language models using the Abstract Meaning Representation (AMR) parsing formalism. |
| Outcome: | The proposed models can reproduce the basic format of AMR, as well as some core event, argument, and modifier structure, but they have virtually no fully accurate parses. |
Exploring BERT’s Sensitivity to Lexical Cues using Tests from Semantic Priming (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using English lexical stimuli, we find that BERT models show "priming" predicting a word with greater probability when the context includes a related word versus an unrelated one. |
| Approach: | They analyze a pre-trained BERT model with tests informed by semantic priming . they find that BERT too shows "priming" predicting a word with greater probability when context includes a related word versus an unrelated one. |
| Outcome: | The proposed model shows a tendency to be distracted by related prime words as context becomes more informative, and lower probability of related words. |
COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models (PLMs) lack robustness in demonstrating simple reasoning, despite having the prerequisite knowledge. |
| Approach: | They propose to test pre-trained language models' ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. |
| Outcome: | The proposed model can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the base of nuanced knowledge representations. |
“No, They Did Not”: Dialogue Response Dynamics in Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | a critical component of competence in language is being able to identify relevant components of an utterance . sensitivity to at-issueness and ellipsis is examined in pre-trained language models . authors find mixed results with respect to capturing full range of dynamics involved in targeting at- issue content . |
| Approach: | They examine sensitivity to at-issueness and ellipsis in pre-trained language models . they find that models show a preference for responses that target main clause content . |
| Outcome: | The proposed models show strong sensitivity to at-issueness and ellipsis dynamics . the results show that the models lack grasp of the dynamics involved in targeting at- issue versus not-at-issue content . |
Can You Follow Me? Testing Situational Understanding for ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have identified certain SU limitations in non-chatbot Large Language models, but the extent and causes of these limitations are not well understood. |
| Approach: | They propose a synthetic environment for SU testing in chat-oriented models . they test models' ability to track and enumerate environment states . |
| Outcome: | The proposed environment allows for controlled and systematic testing of SU in chat-oriented models, and to better understand underlying causes for performance patterns. |
PeTra: A Sparsely Supervised Memory Model for People Tracking (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for coreference resolution scale quadratically with length of text, and are cognitively implausible. |
| Approach: | They propose a memory-augmented neural network called PeTra that tracks entities in its memory slots. |
| Outcome: | The proposed model outperforms a previous memory model while maintaining strong performance. |
Sorting through the noise: Testing robustness of information processing in pre-trained language models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. |
| Approach: | They examine how robustly pre-trained LMs retain and apply relevant context information in the face of distracting content. |
| Outcome: | The proposed models retain and use critical context information in the face of distracting content, while models are susceptible to factors of semantic similarity and word position. |
Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. |
| Approach: | They propose to use in-context examples and instructions to improve LMs' robustness in performing property inheritance. |
| Outcome: | The proposed model can perform non-trivial property inheritance on in-context examples and instructions, but it is inconsistent with the task. |
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)
Copied to clipboard
| Challenge: | a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks . |
| Approach: | They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different . |
| Outcome: | The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability . |
Assessing Composition in Sentence Vector Representations (C18-1)
Copied to clipboard
| Challenge: | opacity of sentence vector representations is a challenge to achieving language understanding . current neural network models are unable to capture meaning information in dense vectors . |
| Approach: | They propose a method that targets compositional meaning information in sentence embeddings with a high degree of precision and control. |
| Outcome: | The proposed method extracts useful information about the different capacities of existing sentences models. |
When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose guidelines for when to implement self-reflection in Large Language Models. |
| Outcome: | The proposed approach improves the reasoning capabilities of Large Language Models under a more stringent evaluation setting, and reduces tendency toward majority voting. |
Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (2023.acl-short)
Copied to clipboard
| Challenge: | Existing language models have demonstrated remarkable improvements in downstream tasks, but it remains unclear to what extent they are consequences of correlation with linguistic heuristics versus robust reasoning about causal relations grounded on understanding of world knowledge. |
| Approach: | They propose to test language models with counterfactual conditionals to test their ability to distinguish hypothetical scenarios from reality. |
| Outcome: | The proposed model overrides real-world knowledge in counterfactual scenarios, but most models are driven by lexical cues. |
Assessing Phrasal Representation and Composition in Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for deep transformers are able to combine word meanings into phrase meanings, but they lack a clear understanding of how they handle complex linguistic inputs. |
| Approach: | They propose to analyze phrasal representations in pre-trained transformers to determine whether they reflect sophisticated composition of phrase meaning. |
| Outcome: | The proposed models are able to combine meaning units into larger units, a phenomenon known as composition, and reflects human understanding of meaning. |
Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current models for document coreference resolution have large memory requirements and quadratic runtime in document length. |
| Approach: | They propose a memory-augmented neural network that tracks only a small number of entities at a time. |
| Outcome: | The proposed model outperforms existing models on OntoNotes and LitBank in memory management and memory management. |