Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Similar Papers
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020.acl-main)
Copied to clipboard
| Challenge: | incorporating syntactic structure into language models has been a challenge since the 1990s. |
| Approach: | They propose to use syntactic information to integrate syntastic structure into neural language models by providing ground truth parse trees as additional training signals. |
| Outcome: | The proposed model achieves lower perplexity and better quality when ground truth parse trees are provided as training signals. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
Mapping Brains with Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | accumulated evidence for brain and language model activations remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism. |
| Approach: | They examine the evidence accumulated by 30 studies spanning 10 datasets and 8 metrics to determine whether there is any overlap between brain and language model activations. |
| Outcome: | The findings suggest that representations extracted from NLP models can (partially) explain the signal found in neural data. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that models encode syntactic information redundantly . this allows researchers to boost models' performance by injecting syntaktic information into embeddings . |
| Approach: | They propose a new probe design that guides probes to consider all syntactic information present in embeddings. |
| Outcome: | The proposed model improves performance by injecting syntactic information into models. |
Targeted Syntactic Evaluation of Language Models (D18-1)
Copied to clipboard
| Challenge: | Recent advances have led to an explosion of neural network-based LM architectures. |
| Approach: | They propose to supplement perplexity with a metric that assesses whether a language model can predict the grammatical sentence more accurately than an ungrammatically-based model. |
| Outcome: | The proposed model performed poorly on many of the constructions. |
Language Model Evaluation Beyond Perplexity (2021.acl-long)
Copied to clipboard
| Challenge: | a nascent literature on probing language models has focused on studying linguistic phenomena. |
| Approach: | They propose a framework for evaluating the fit of language models to natural language tendencies. |
| Outcome: | The proposed framework evaluates language models to the tendencies of natural language . it shows that the models learn only a subset of the tendancies considered . |
Lexicosyntactic Inference in Neural Models (D18-1)
Copied to clipboard
| Challenge: | lexicosyntactic inferences are triggered by surprising aspects of the syntactical context that a word occurs in. |
| Approach: | They build a factuality judgment dataset for English clause-embedding verbs in various syntactic contexts and use it to probe the behavior of current state-of-the-art neural systems. |
| Outcome: | The proposed model makes systematic errors that are visible through the lens of factuality prediction. |