| Challenge: | Neural networks are the backbone of modern stateof-the-art natural language processing systems. |
| Approach: | They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. |
| Outcome: | The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages. |
Similar Papers
Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how and to what extent do these representations encode a particular linguistic phenomenon. |
| Approach: | They propose a new information-theoretic probe, Bird’s Eye, which detects if and how representations encode the information in contextualized text representations. |
| Outcome: | The proposed method estimates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word representations. |
An information theoretic view on selecting linguistic probes (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP tasks require a question of how much linguistic knowledge is encoded in neural networks. |
| Approach: | They propose to use diagnostic classifiers to perform supervised classification from internal representations. |
| Outcome: | Empirically, the two proposed criteria lead to results that agree with each other. |
A Tale of a Probe and a Parser (2020.acl-main)
Copied to clipboard
| Challenge: | researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees . |
| Approach: | They compare a structural probe to a more traditional parser with a lightweight parameterisation. |
| Outcome: | The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points . |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
Where Does Linguistic Information Emerge in Neural Language Models? Measuring Gains and Contributions across Layers (2022.coling-1)
Copied to clipboard
| Challenge: | Probing studies have explored where in neural language models linguistic information is located . standard approach is to focus on the layers whose representations give the highest performance on probing tasks . |
| Approach: | They propose a method that asks where task-relevant information emerges in the model by focusing on the layers that give the highest performance. |
| Outcome: | The proposed method confirms the expected ordering only for one of the pairs, indicating that the features that contribute the most to probing tasks are not as high-level as global metrics suggest. |
A Bayesian Framework for Information-Theoretic Probing (2021.emnlp-main)
Copied to clipboard
| Challenge: | a recent paper suggests that probing should be seen as approximating a mutual information. |
| Approach: | They propose a Bayesian mutual information framework that probes probing representations from the perspective of Bayes' agents. |
| Outcome: | The proposed framework allows for more intuitive results in scenarios with finite data. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
Information-Theoretic Probing with Minimum Description Length (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. |
| Approach: | They propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). |
| Outcome: | The proposed method agrees in results and is more informative and stable than the standard probes. |
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing (2021.naacl-main)
Copied to clipboard
| Challenge: | a study of neural language models shows that syntactic probes do not properly isolate syntax. |
| Approach: | They show that syntactic probes do not properly isolate syntax . they train two probes trained on normal data and find they perform worse . |
| Outcome: | The proposed method outperforms the baseline models on the most popular models, but their lead is reduced by 53%. |