What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
Similar Papers
Empirical Linguistic Study of Sentence Embeddings (P19-1)
Copied to clipboard
| Challenge: | a new method of analysing sentence embeddings shows that linguistic information is retained in the vector representations of sentences. |
| Approach: | They propose a method of analysing the content of sentence embeddings based on probing tasks and contrasting languages. |
| Outcome: | The proposed method is based on probing tasks and classification datasets for two contrasting languages. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Exploring Semantic Properties of Sentence Embeddings (P18-2)
Copied to clipboard
| Challenge: | Neural vector representations are ubiquitous throughout all subfields of natural language processing. |
| Approach: | They propose a framework that generates triplets of sentences to explore how changes in the syntactic structure or semantics of a given sentence affect their similarity. |
| Outcome: | The proposed framework generates triplets of sentences to explore how changes in the syntactic structure or semantics of a given sentence affect the similarities obtained between their embeddings. |
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case (2020.coling-main)
Copied to clipboard
| Challenge: | Semantic embeddings have advanced the state of the art for natural language processing tasks . but their inner workings are poorly understood and there is a shortage of analysis tools . |
| Approach: | They propose to extend visual-semantic embeddings to multimodal domains by defining probing tasks for embeddable image-caption pairs and testing them with classifiers. |
| Outcome: | The proposed probing tasks show up to 16% more accurate on visual-semantic embeddings compared to unimodal embedders . the proposed extensions to multimodal domains have been lauded as promising in natural language processing . |
Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Neural relation extraction models capture linguistic and semantic properties of the input, a recent study shows. |
| Approach: | They introduce 14 probing tasks targeting linguistic properties relevant to RE . they add contextualized word representations to enhance probing performance . |
| Outcome: | The proposed models achieve state-of-the-art on two datasets, TACRED and SemEval 2010 Task 8 . they show that the models capture linguistic and semantic properties relevant to the downstream task . |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)
Copied to clipboard
| Challenge: | Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations. |
| Approach: | They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis . |
| Outcome: | The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures. |
Semantic Geometry of Sentence Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sentence embeddings are central to natural language processing, but their internal features are not interpretable and users lack fine-grained control for downstream tasks. |
| Approach: | They propose a formal framework to characterize the organization of features in sentence embeddings . they show how they can be composed to capture richer semantic structures . |
| Outcome: | The proposed method can be used to capture richer semantic structures. |
PWESuite: Phonetic Word Embeddings and Tasks They Facilitate (2024.lrec-main)
Copied to clipboard
Vilém Zouhar, Kalvin Chang, Chenxuan Cui, Nate B. Carlson, Nathaniel Romney Robinson, Mrinmaya Sachan, David R. Mortensen
| Challenge: | Existing word embedding methods overlook phonetic information that is crucial for many tasks. |
| Approach: | They propose three methods that use articulatory features to build phonetically informed word embeddings. |
| Outcome: | The proposed methods improve word retrieval and correlation with sound similarity and on rhyme and cognate detection tasks. |