| Challenge: | Probing is a method of investigating the encoding of knowledge in contextual representations. |
| Approach: | They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe. |
| Outcome: | The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages. |
Similar Papers
A Structural Probe for Finding Syntax in Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Approach: | They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Outcome: | The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry. |
A Tale of a Probe and a Parser (2020.acl-main)
Copied to clipboard
| Challenge: | researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees . |
| Approach: | They compare a structural probe to a more traditional parser with a lightweight parameterisation. |
| Outcome: | The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points . |
Introducing Orthogonal Constraint in Structural Probes (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on interpreting pre-trained models' representations and analyzing their structures. |
| Approach: | They propose a new type of structural probing where a linear projection is decomposed into two types. |
| Outcome: | The proposed method is tested on two novel tasks and shows that lexical and syntactic information is separated in the representations. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
Probing for Constituency Structure in Neural Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using standard probing techniques, we examine whether contextual neural language models implicitly learn syntactic structure. |
| Approach: | They investigate to which extent contextual neural language models implicitly learn syntactic structure. |
| Outcome: | The proposed model is able to represent constituents of different categories within the neuron activations of a LM such as RoBERTa with high performance even on manipulated data. |
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing (2021.naacl-main)
Copied to clipboard
| Challenge: | a study of neural language models shows that syntactic probes do not properly isolate syntax. |
| Approach: | They show that syntactic probes do not properly isolate syntax . they train two probes trained on normal data and find they perform worse . |
| Outcome: | The proposed method outperforms the baseline models on the most popular models, but their lead is reduced by 53%. |
Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on whether multilingual embeddings can be aligned in a shared space across languages are lacking. |
| Approach: | They propose to learn a projection based on monolingual annotated datasets and evaluate syntactic and lexical information encoded in a shared cross-lingual embedding space. |
| Outcome: | The proposed model can be used to learn representations for languages with low resources. |
Does My Representation Capture X? Probe-Ably (2021.acl-demo)
Copied to clipboard
| Challenge: | Probing (or diagnostic classification) has become a popular strategy for investigating whether a given set of intermediate features is present in the representations of neural models. |
| Approach: | They propose to use an extendable probing framework to automate the application of probing methods to the user’s inputs. |
| Outcome: | The proposed framework automates the application of probing methods to the user’s inputs. |
Information-Theoretic Probing for Linguistic Structure (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are the backbone of modern stateof-the-art natural language processing systems. |
| Approach: | They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. |
| Outcome: | The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages. |
What Does Parameter-free Probing Really Uncover? (2024.acl-short)
Copied to clipboard
| Challenge: | Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels. |
| Approach: | They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT. |
| Outcome: | The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory. |