| Challenge: | Existing studies on supervised models to predict properties from representations have shown high accuracy on a range of linguistic tasks. |
| Approach: | They propose control tasks which associate word types with random outputs to complement linguistic tasks by construction . they find that popular probes on ELMo representations are not selective . |
| Outcome: | The proposed tasks associate word types with random outputs to complement linguistic tasks. |
Similar Papers
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
An information theoretic view on selecting linguistic probes (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP tasks require a question of how much linguistic knowledge is encoded in neural networks. |
| Approach: | They propose to use diagnostic classifiers to perform supervised classification from internal representations. |
| Outcome: | Empirically, the two proposed criteria lead to results that agree with each other. |
Information-Theoretic Probing for Linguistic Structure (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are the backbone of modern stateof-the-art natural language processing systems. |
| Approach: | They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. |
| Outcome: | The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages. |
Probing Task-Oriented Dialogue Representation from Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find out which model has the most informative representation for task-oriented dialogue tasks. |
| Approach: | They propose a supervised classifier probe and unsupervised mutual information probe to investigate the mutual dependence between a real clustering and a representation clustering. |
| Outcome: | The proposed model is a supervised classifier probe and unsupervised mutual information probe. |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
Improving Syntactic Probing Correctness and Robustness with Control Tasks (2023.acl-short)
Copied to clipboard
Weicheng Ma, Brian Wang, Hefan Zhang, Lili Wang, Rolando Coto-Solano, Saeed Hassanpour, Soroush Vosoughi
| Challenge: | Syntactic probing methods are biased by the PLMs’ memorization of common word co-occurrences, even if they do not form syntactical relations. |
| Approach: | They propose to use random word substitution and random label matching to reduce these biases and improve the robustness of syntactic probing methods. |
| Outcome: | The proposed tasks improve probing results and consistency between probing methods and make them more generalizable to unseen text domains. |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Neural relation extraction models capture linguistic and semantic properties of the input, a recent study shows. |
| Approach: | They introduce 14 probing tasks targeting linguistic properties relevant to RE . they add contextualized word representations to enhance probing performance . |
| Outcome: | The proposed models achieve state-of-the-art on two datasets, TACRED and SemEval 2010 Task 8 . they show that the models capture linguistic and semantic properties relevant to the downstream task . |
Low-Complexity Probing via Finding Subnetworks (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to probing neural networks for linguistic properties are to train a shallow multi-layer perceptron (MLP) on top of the model's internal representations. |
| Approach: | They propose a subtractive pruning-based probe where they find an existing subnetwork that performs the linguistic task of interest. |
| Outcome: | The proposed probe achieves higher accuracy on pre-trained models and lower accuracy on random models, and better learning on its own. |