| Challenge: | Modern NLP systems rely on word embeddings as input units to encode statistical semantic and syntactic properties of words. |
| Approach: | They propose to use discrete cosine transform to compress word sequences in order-preserving manner. |
| Outcome: | The proposed model preserves syntactic information in semantic probing tasks . it is comparable to vector averaging but mediocre in performance. |
Similar Papers
Discrete Cosine Transform as Universal Sentence Encoder (2021.acl-short)
Copied to clipboard
| Challenge: | Modern sentence encoders capture underlying linguistic characteristics of words . Discrete Cosine Transform (DCT) is an efficient alternative to averaging . |
| Approach: | They propose to use a Discrete Cosine Transform to generate universal sentence representations in different languages. |
| Outcome: | The proposed model captures the underlying syntactic characteristics of a given text without compromising practical efficiency. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
EigenSent: Spectral sentence embeddings using higher-order Dynamic Mode Decomposition (P19-1)
Copied to clipboard
| Challenge: | Existing methods for calculating word embeddings use weighted averages or learn word representations, but this is not the case. |
| Approach: | They propose to use spectral methods of signal representation and summarization to construct word-sequence embeddings in unsupervised fashion by using a spectrally-based decomposition and signal summarizing technique on text. |
| Outcome: | The proposed method can summarize transitions in a sequence of words and generate an embedding that can represent well the sequence itself. |
More Discriminative Sentence Embeddings via Semantic Graph Smoothing (2024.eacl-short)
Copied to clipboard
| Challenge: | Text categorization is a natural language processing task that involves arranging texts into coherent groups based on their content. |
| Approach: | They propose to use semantic graph smoothing to enhance sentence embeddings from pretrained models to improve results for supervised and unsupervised document categorization tasks. |
| Outcome: | The proposed method improves sentences embeddings for supervised and unsupervised document categorization tasks. |
Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for word embedding compression are limited . word embeds have a considerable size and need to be compressed to deploy on edge devices . |
| Approach: | They propose a block-wise low-rank approximation method for word embedding called GroupReduce . they propose 'frequency-inverse document frequency method' and a differentiable method for weighting . |
| Outcome: | The proposed algorithm more effectively finds word weights than competitors in most cases. |
Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform (2024.findings-acl)
Copied to clipboard
| Challenge: | Discrete Wavelet Transforms (DWT) can be applied to NLP capturing a variety of linguistic and semantic properties. |
| Approach: | They propose to use Discrete Wavelet Transforms to analyze word and sentence embeddings . they show that DWT can reduce dimensionality of embeddables by 50-93% . |
| Outcome: | The proposed paradigm reduces embeddings' dimensionality by 50-93% while maintaining their quality. |
Learning Compressed Sentence Representations for On-Device Text Processing (P19-1)
Copied to clipboard
Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang, Asli Celikyilmaz, Lawrence Carin
| Challenge: | Existing methods for learning sentence embeddings assume they are continuous and real-valued. |
| Approach: | They propose four different strategies to transform continuous and generic sentence embeddings into a binarized form while preserving their rich semantic information. |
| Outcome: | The proposed methods reduce storage requirements by over 98% and improve performance on downstream tasks. |
Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation (N19-1)
Copied to clipboard
| Challenge: | a novel word embedding representation for text documents is proposed . the method is based on the Vector of Locally-Aggregated Descriptors used for image representation . |
| Approach: | They propose a novel representation for text documents based on aggregating word embedding vectors into document embeddables. |
| Outcome: | The proposed representation improves on the Movie Review data set and is 10% better than the state-of-the-art representation. |
Towards Lossless Encoding of Sentences (P19-1)
Copied to clipboard
| Challenge: | Existing methods for encoding text into lossless representations focus on performing well on downstream tasks and are unable to reconstruct original sequence from learned embedding. |
| Approach: | They propose a lossless method for encoding long sequences of texts into feature rich representations by recursive autoencoding. |
| Outcome: | The proposed method performs well on sentiment analysis and sentiment classification tasks. |
Adaptive Compression of Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks. |
| Approach: | They propose a method that uses a code-book approach to represent words as discrete codes such as (8, 5, 2, 4). |
| Outcome: | The proposed method makes the highly compressed word embeddings without hurting the task accuracy. |