Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Standard language models employ unique, monolithic embeddings for each token, limiting their ability to capture multifaceted meanings. |
| Approach: | They propose a compositional structure that accumulates diverse semantic facets for tokens . they apply this representational scheme to standard transformer architectures and a biomedical domain benchmark . |
| Outcome: | The proposed representational scheme achieves extreme compression in embedding parameters while maintaining >95% task performance relative to the base model. |
Similar Papers
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that sequence-to-sequence (seq2sequ) models struggle with compositional generalization (CG) a crucial property of human language learning is its compositional globalization (GC), the algebraic ability to understand and produce a potentially infinite number of novel combinations from known components. |
| Approach: | They propose a sequence-to-sequence (seq2sequ) extension which learns to compose representations of different encoder layers dynamically for different tasks. |
| Outcome: | The proposed model achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of the proposed model. |
Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory. |
| Approach: | They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules. |
| Outcome: | The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. |
Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Speech Language Models (LSLMs) typically operate at high token rates to ensure acoustic fidelity, yet this results in sequence lengths that exceed the underlying semantic content, incurring prohibitive inference costs. |
| Approach: | They propose a token-based token merging mechanism that uses a training-free token pooling mechanism to reduce prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
| Outcome: | The proposed method reduces prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
How to Leverage Digit Embeddings to Represent Numbers? (2025.coling-main)
Copied to clipboard
| Challenge: | Existing numerical reasoning models struggle to understand numbers, despite simple generalisations. |
| Approach: | They propose to use mathematical priors to compute digit embeddings and explicitly incorporate them into transformer models by adding a special token to the input embedded digits or introducing an additional loss function to enhance correct predictions. |
| Outcome: | The proposed method is compatible with any pretrained model and easy to implement. |
Enhanced Word Representations for Bridging Anaphora Resolution (N18-2)
Copied to clipboard
| Challenge: | Existing word representations do not capture semantic similarity for bridging anaphora resolution. |
| Approach: | They propose to use word embeddings to capture semantic similarity by exploring syntactic structure of noun phrases. |
| Outcome: | The proposed model achieves 30% of accuracy for bridging anaphora resolution on ISNotes corpus. |
From Characters to Tokens: Dynamic Grouping with Hierarchical BPE (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization methods suffer from inefficiencies in representing rare words and require large embedding matrices. |
| Approach: | They propose a dynamic character grouping method that leverages existing BPE tokenization without adding additional models. |
| Outcome: | The proposed method matches or exceeds existing entropy- and whitespace-based patching strategies while maintaining a compact vocabulary. |
Compact Token Representations with Contextual Quantization for Efficient Document Re-ranking (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work has adopted a late interaction architecture with pre-computed contextual token representations at the cost of a large online storage. |
| Approach: | They propose to decouple document-specific and document-independent ranking contributions during codebook-based compression for effective online decompression and embedding composition for better search relevance. |
| Outcome: | The proposed model achieves high relevance and space efficiency with minimal computational cost and low computational complexity. |
On the Way to Lossless Compression of Language Transformers: Exploring Cross-Domain Properties of Quantization (2024.lrec-main)
Copied to clipboard
Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, Egor Anatolievich Filimonov
| Challenge: | Modern Natural Language Processing models have a huge capacity, but this makes it difficult to employ. |
| Approach: | They propose a method to quantize at least 95% of Transformer weights without access to task-specific data so the drop in performance does not exceed 0.02%. |
| Outcome: | The proposed method quantizes 95% of Transformer weights and corresponding activations to INT8 without access to task-specific data so the drop in performance does not exceed 0.02%. |