| Challenge: | Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Approach: | They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Outcome: | The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry. |
Similar Papers
A Tale of a Probe and a Parser (2020.acl-main)
Copied to clipboard
| Challenge: | researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees . |
| Approach: | They compare a structural probe to a more traditional parser with a lightweight parameterisation. |
| Outcome: | The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points . |
A Non-Linear Structural Probe (2021.naacl-main)
Copied to clipboard
| Challenge: | Probing is a method of investigating the encoding of knowledge in contextual representations. |
| Approach: | They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe. |
| Outcome: | The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages. |
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020.acl-main)
Copied to clipboard
| Challenge: | incorporating syntactic structure into language models has been a challenge since the 1990s. |
| Approach: | They propose to use syntactic information to integrate syntastic structure into neural language models by providing ground truth parse trees as additional training signals. |
| Outcome: | The proposed model achieves lower perplexity and better quality when ground truth parse trees are provided as training signals. |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
Probing for Constituency Structure in Neural Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using standard probing techniques, we examine whether contextual neural language models implicitly learn syntactic structure. |
| Approach: | They investigate to which extent contextual neural language models implicitly learn syntactic structure. |
| Outcome: | The proposed model is able to represent constituents of different categories within the neuron activations of a LM such as RoBERTa with high performance even on manipulated data. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on whether multilingual embeddings can be aligned in a shared space across languages are lacking. |
| Approach: | They propose to learn a projection based on monolingual annotated datasets and evaluate syntactic and lexical information encoded in a shared cross-lingual embedding space. |
| Outcome: | The proposed model can be used to learn representations for languages with low resources. |
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on enhancing existing models with the primary objective of improving downstream performance on various NLP tasks. |
| Approach: | They propose to use BERT to encode meaningful knowledge in token representations to explain probing results. |
| Outcome: | The proposed model can detect syntactic and semantic abnormalities and distinguish between grammatical number and tense subspaces. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |
What Does Parameter-free Probing Really Uncover? (2024.acl-short)
Copied to clipboard
| Challenge: | Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels. |
| Approach: | They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT. |
| Outcome: | The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory. |