| Challenge: | a new analysis method for contextualized representations is proposed . contextualization boosts reconstructability of tokens close to the token being reconstructed . |
| Approach: | They propose a method for contextualized representations based on reconstruction probabilities in masked language models. |
| Outcome: | The proposed method compares reconstruction probabilities of tokens in masked language models . it finds that contextualization boosts reconstructability of token that are close to the token being reconstructed . |
Similar Papers
Probing Contextual Language Models for Common Ground with Visual Representations (2021.naacl-main)
Copied to clipboard
| Challenge: | Contextual language models have attracted great interest in probing what is encoded in their representations. |
| Approach: | They propose a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. |
| Outcome: | The proposed model outperforms text-only language models in instance retrieval, but underperform humans. |
Probing for Constituency Structure in Neural Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using standard probing techniques, we examine whether contextual neural language models implicitly learn syntactic structure. |
| Approach: | They investigate to which extent contextual neural language models implicitly learn syntactic structure. |
| Outcome: | The proposed model is able to represent constituents of different categories within the neuron activations of a LM such as RoBERTa with high performance even on manipulated data. |
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)
Copied to clipboard
| Challenge: | a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks . |
| Approach: | They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different . |
| Outcome: | The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability . |
Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations. |
| Approach: | They propose a representation learning approach that uses embeddings as anchors to model contextual representations. |
| Outcome: | The proposed model achieves 5x speedup and 1.2 points average improvement over MLM. |
What Does Parameter-free Probing Really Uncover? (2024.acl-short)
Copied to clipboard
| Challenge: | Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels. |
| Approach: | They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT. |
| Outcome: | The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory. |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
C3D: Enhancing LLM Reasoning via Confidence-Guided Contrastive Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are prone to distraction by contextual information during reasoning tasks. |
| Approach: | They propose a decoding method that uses predicted logits to estimate the model's confidence. |
| Outcome: | The proposed method reveals how the model dynamically activates and adjusts its consideration of each premise as reasoning progresses. |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
Probing as Quantifying Inductive Bias (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextual representations have led to performance improvements on downstream tasks. |
| Approach: | They propose a Bayesian framework that quantifies the amount of inductive bias that the representations encode on a specific task. |
| Outcome: | The proposed framework alleviates many problems found in probing and can offer better inductive bias than BERT. |