| Challenge: | opacity of sentence vector representations is a challenge to achieving language understanding . current neural network models are unable to capture meaning information in dense vectors . |
| Approach: | They propose a method that targets compositional meaning information in sentence embeddings with a high degree of precision and control. |
| Outcome: | The proposed method extracts useful information about the different capacities of existing sentences models. |
Similar Papers
Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing computational metric to quantify extent of meaning composition is lacking . despite extensive neurolinguistic research, understanding how meaning is constructed is difficult . |
| Approach: | They propose a computational metric to quantify the degree of meaning composition . they use key-value memory interpretation of transformer feed-forward network blocks to study meaning composition. |
| Outcome: | Experimental results show that the metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words, suggesting multifaceted nature of meaning composition during human sentence comprehension. |
Sentence-Level Content Planning and Style Specification for Neural Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in text generation systems often produce incoherent and unfaithful outputs . a novel automated text generation system takes into account content selection, text planning, and surface realization. |
| Approach: | They propose an end-to-end trained two-step text generation model that considers sentence-level content planners and language styles. |
| Outcome: | The proposed model outperforms competing models in three domains with diverse topics and varying language styles. |
Exploring Semantic Properties of Sentence Embeddings (P18-2)
Copied to clipboard
| Challenge: | Neural vector representations are ubiquitous throughout all subfields of natural language processing. |
| Approach: | They propose a framework that generates triplets of sentences to explore how changes in the syntactic structure or semantics of a given sentence affect their similarity. |
| Outcome: | The proposed framework generates triplets of sentences to explore how changes in the syntactic structure or semantics of a given sentence affect the similarities obtained between their embeddings. |
Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings (2023.eacl-main)
Copied to clipboard
| Challenge: | Past work on sentence embedding models faces issues determining the causal impact of implicit syntax representations. |
| Approach: | They construct a neural module net based on a transformer model and train it end-to-end to approximate the sentence’s embedding. |
| Outcome: | The proposed model captures whether syntax is a strong model of its compositional ability. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
Compositional Generalization for Data-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Data-to-text generation models can be used to generate textual descriptions from structured data . despite advances, systems struggle when confronted with unseen combinations of predicates . |
| Approach: | They propose a data-to-text generation model that addresses compositional generalization by clustering predicates into groups. |
| Outcome: | The proposed model outperforms T5-baselines in all evaluation metrics. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021.acl-short)
Copied to clipboard
| Challenge: | Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. |
| Approach: | They propose to convert a natural language sequence-to-sequence dataset into a classification dataset that requires compositional generalization. |
| Outcome: | The proposed model can generalize compositionally by providing hints on the structure of the input. |
Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings do not capture the semantic similarity of sentences. |
| Approach: | They propose a framework that integrates compositional sentence operations into the embedding space and optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddables. |
| Outcome: | The proposed framework improves the interpretability of sentence embeddings on four textual generation tasks while maintaining strong performance on traditional semantic similarity tasks. |