Papers by Roger Levy
Language model acceptability judgements are not always robust to context (2023.acl-long)
Copied to clipboard
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
| Challenge: | a recent study found that models prefer acceptable inputs over acceptable ones. |
| Approach: | They find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but unstable when contexts match the test stimuli in syntactic structure. |
| Outcome: | The proposed model performance improves when contexts match syntactic structure, and declines when they are unacceptable. |
The neural dynamics of word recognition and integration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a computational model of word recognition, listeners combine expectations about upcoming content with incremental sensory evidence. |
| Approach: | They fit this model to scalp EEG signals recorded as subjects passively listened to a fictional story and found that words require more than 150 ms of input to be recognized. |
| Outcome: | The proposed model formalizes this perceptual process in Bayesian decision theory and reveals distinct neural processing of words depending on whether or not they can be quickly recognized. |
Language Learning and Processing in People and Machines (N19-5)
Copied to clipboard
| Challenge: | This tutorial introduces different stages of language acquisition and their parallel problems in NLP. |
| Approach: | This tutorial introduces different stages of language acquisition and their parallel problems in NLP. |
| Outcome: | This tutorial introduces different stages of language acquisition and their parallel problems in NLP. |
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not investigated the relationship between a token's frequency in the training corpus and syntactic properties models learn about it. |
| Approach: | They develop controlled experiments that probe models’ syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. |
| Outcome: | The proposed models can make syntactic generalizations for tokens seen as few as two times during training and transfer them to transformed contexts. |
STARC: Structured Annotations for Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | STARC is an annotation framework for reading comprehension with multiple choice questions . existing annotation frameworks have limited support for reading behavior analyses . |
| Approach: | They propose a new annotation framework for assessing reading comprehension with multiple choice questions . they use a span ablation dataset to demonstrate that it can be leveraged for a key new application . |
| Outcome: | The proposed framework can be leveraged for a key new application for SAT-like reading comprehension materials. |
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that models encode syntactic information redundantly . this allows researchers to boost models' performance by injecting syntaktic information into embeddings . |
| Approach: | They propose a new probe design that guides probes to consider all syntactic information present in embeddings. |
| Outcome: | The proposed model improves performance by injecting syntactic information into models. |
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models (2020.acl-demos)
Copied to clipboard
| Challenge: | SyntaxGym is an online platform and open-source framework for targeted syntactic evaluation of neural network language models. |
| Approach: | They propose to make targeted syntactic evaluations accessible to both experts in NLP and linguistics and reproducible across computing environments. |
| Outcome: | The proposed framework is reproducible across computing environments and standardized following the norms of psycholinguistic experimental design. |
Assessing Language Proficiency from Eye Movements in Reading (N18-1)
Copied to clipboard
| Challenge: | a novel approach to determine second language proficiency uses behavioral traces of eye movements during reading . over 1.5 billion people are learning English as a second language worldwide . traditional approaches to language proficiency testing have several drawbacks, including the fact that they are typically prepared manually and require extensive resources for test development . |
| Approach: | They propose a method which uses behavioral traces of eye movements during reading to determine learners’ second language proficiency. |
| Outcome: | The proposed approach correlates with standardized English proficiency tests and is validated by eyetracking with eye movements from other readers. |
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior art aims to uncover meaningful properties within model representations, but it is unclear how faithfully such probes portray information that the models actually use. |
| Approach: | They propose a technique for generating counterfactual embeddings within models . they produce evidence that some models use a tree-distancelike representation of syntax . |
| Outcome: | The proposed technique produces evidence that some models use tree-distancelike representations of syntax in downstream prediction tasks. |
Revisiting the Uniform Information Density Hypothesis (2021.emnlp-main)
Copied to clipboard
| Challenge: | The uniform information density hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. |
| Approach: | They propose to test the hypothesis by using reading time and acceptability data to examine the effect of surprisal on language comprehension and acceptabilities. |
| Outcome: | The proposed hypothesis makes predictions about language comprehension and linguistic acceptability . |
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. |
| Approach: | They train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on Mandarin Chinese datasets. |
| Outcome: | The proposed models learn aspects of Mandarin Chinese grammar that assess syntactic and semantic relationships. |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |
A Cross-Linguistic Pressure for Uniform Information Density in Word Order (2023.tacl-1)
Copied to clipboard
Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, Roger Levy
| Challenge: | a recent study has compared real and counterfactual word orders, but one functional pressure has been overlooked . a study of 10 typologically diverse languages shows that real word orders have greater uniformity than reverse word orders . |
| Approach: | They propose to test whether a pressure for UID may have influenced word order patterns cross-linguistically. |
| Outcome: | The proposed model shows that real orders have greater uniformity than reverse orders among SVO languages. |
A Targeted Assessment of Incremental Processing in Neural Language Models and Humans (2021.acl-long)
Copied to clipboard
| Challenge: | Using by-word reaction time data, we compare incremental processing in humans and neural language models across a range of structural phenomena. |
| Approach: | They propose to scale up incremental processing in humans and language models by collecting by-word reaction time data for 16 different syntactic test suites. |
| Outcome: | The proposed model outputs match human and model accuracy scores, but underpredict the difference in magnitude of incremental processing difficulty between grammatical and ungrammatically-spaced sentences. |
Linking artificial and human neural representations of language (D19-1)
Copied to clipboard
| Challenge: | a pre-trained BERT architecture is used to fine-tune sentence encoding models on a variety of natural language understanding (NLU) tasks. |
| Approach: | They compare sentence encoding models with fMRI-based fMR predictions of the sentence . they use a pre-trained BERT architecture as a baseline and fine-tune it on a variety of natural language understanding (NLU) tasks. |
| Outcome: | The proposed model does not yield significant improvements in brain decoding performance on the natural language understanding (NLU) tasks. |
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023.emnlp-main)
Copied to clipboard
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, Roger Levy
| Challenge: | Logical reasoning is an important task for artificial intelligence, says a new study . many prompting-based strategies to enable large language models fail in subtle and unpredictable ways. |
| Approach: | They propose to reformulate logical reasoning tasks by leveraging large language models . they use a modular neurosymbolic programming approach to translate premises and conclusions from natural language to logic . |
| Outcome: | The proposed approach outperforms open-source models on FOLIO and ProofWriter while showing distinct failure modes. |
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
Flexible Generation from Fragmentary Linguistic Input (2022.acl-long)
Copied to clipboard
| Challenge: | dominant paradigm for high-performance models in novel language tasks is direct specialization via training from scratch or fine-tuning large pre-trained models. |
| Approach: | They propose a new model that makes it possible to infer human behavior through basic computational motifs. |
| Outcome: | The proposed model outperforms direct-specialization models in three evaluations and performs comparable to human models. |
Structural Guidance for Transformer Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer language models have proven remarkably successful in learning generic transferable linguistic representations without resorting to data intensive pre-training. |
| Approach: | They propose to combine a generative parsing and a structural scaffolding idea to guide the model's representation via additional structure loss that separates the incremental constituency parse. |
| Outcome: | The proposed models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments, and provide useful linguistic representations that benefit a wide range of downstream tasks. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
Comparing Theories of Speaker Choice Using a Model of Classifier Production in Mandarin Chinese (N18-1)
Copied to clipboard
| Challenge: | Existing studies show that optional reductions are sensitive to contextual predictability . unclear whether speaker choices are driven by audience design or to facilitate production . |
| Approach: | They argue that Uniform Information Density and availability-based production make opposite predictions regarding the predictability of upcoming material and speaker choices. |
| Outcome: | The proposed model shows that speaker choices support availability-based production account, not the UID hypothesis. |
Expectations over Unspoken Alternatives Predict Pragmatic Inferences (2023.tacl-1)
Copied to clipboard
| Challenge: | Scalar inferences (SI) are a signature example of how humans interpret language based on unspoken alternatives. |
| Approach: | They propose to use context-driven expectations to explain scale-based inferences . they find that expectedness of a strong scalemate captures SI rates within and across scales - but only under meaning-based view of alternatives. |
| Outcome: | The proposed model captures SI rates by expectedness of a strong scalemate as an alternative, but only under a meaning-based view of alternatives. |
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study (D19-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of neural models to compute and employ phrase-level features attached to a set of words, such as subject number or whquestion words. |
| Approach: | They examine whether models can represent constituent-level features, using coordinated noun phrases as a case study. |
| Outcome: | The proposed model can combine gender and gender features to drive downstream expectations, while having less success with gender agreement. |
Analyzing Wrap-Up Effects through an Information-Theoretic Lens (2022.acl-short)
Copied to clipboard
| Challenge: | a lack of studies targeting naturalistic sentence-final reading behavior is likely to explain the lack of data on reading time (RT) data is omitted due to the confounding factors introduced by so-called "wrap-up effects" |
| Approach: | They propose to look for a link between “wrap-up effects” and information theoretic quantities such as word and context information content. |
| Outcome: | The proposed model omits data on words at the end of sentences or clauses to control for the confounding factors introduced by wrap-up effects. |
Prompting is not a substitute for probability measurements in large language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompting is a dominant method for evaluating the linguistic knowledge of large language models (LLMs). |
| Approach: | They compare metalinguistic prompting and direct probability measurements as ways of measuring LLMs’ linguistic knowledge. |
| Outcome: | The results show that the results relying on metalinguistic prompts cannot be taken as conclusive evidence that an LLM lacks a particular linguistic generalization. |
Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammars (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that context-free grammars are not natural for modeling discontinuous language phenomena such as extrapositions and cross-serial dependencies. |
| Approach: | They propose a grammar induction approach with mildly context-sensitive grammars for unsupervised discontinuous parsing. |
| Outcome: | Experiments on German and Dutch show that the proposed grammar induction method is beneficial for unsupervised parsing. |
Probing for Incremental Parse States in Autoregressive Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on autoregressive language models shows impressive command of syntax . implicit incremental syntactic inferences underlie next-word predictions . |
| Approach: | They propose a probe for extracting incomplete syntactic structure from autoregressive language models. |
| Outcome: | The proposed probes can predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior. |