Word Embedding Binarization with Semantic Information Preservation (2020.coling-main)
Copied to clipboard
| Challenge: | Word and Sentence embeddings are one of the most common starting points of any NLP task. |
| Approach: | They propose a way to convert word embedding to binary representation to reduce overall size . they propose different approaches suitable for different downstream tasks based on contextual and semantic information. |
| Outcome: | The proposed method reduces the size of the embedding while keeping the semantic and relational knowledge intact. |
Similar Papers
Learning Compressed Sentence Representations for On-Device Text Processing (P19-1)
Copied to clipboard
Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang, Asli Celikyilmaz, Lawrence Carin
| Challenge: | Existing methods for learning sentence embeddings assume they are continuous and real-valued. |
| Approach: | They propose four different strategies to transform continuous and generic sentence embeddings into a binarized form while preserving their rich semantic information. |
| Outcome: | The proposed methods reduce storage requirements by over 98% and improve performance on downstream tasks. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Deconstructing word embedding algorithms (2020.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are reliable feature representations of words used in many NLP tasks today. |
| Approach: | They propose to deconstruct Word2vec, GloVe and others into a common form . they propose to generalize several word embedding algorithms into . a low rank embedder framework is proposed to generalise the algorithms into one common form. |
| Outcome: | The proposed framework can be used to make word embeddings more performant. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly. |
| Approach: | They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution. |
| Outcome: | The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Adaptive Compression of Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks. |
| Approach: | They propose a method that uses a code-book approach to represent words as discrete codes such as (8, 5, 2, 4). |
| Outcome: | The proposed method makes the highly compressed word embeddings without hurting the task accuracy. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
EmByte: Decomposition and Compression Learning for Small yet Private NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EMBYTE is a byte-level tokenization model that reduces embedding parameters by up to 94% . it is also resilient to privacy threats such as gradient inversion attacks . |
| Approach: | EMBYTE is a byte-level tokenization model that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. |
| Outcome: | EMBYTE achieves substantial embedding compression while preserving accuracy and enhancing privacy. |
Explaining Word Embeddings via Disentangled Representation (2020.aacl-main)
Copied to clipboard
| Challenge: | Disentangled representations are known to represent interpretable factors in separated dimensions. |
| Approach: | They propose to transform dense word vectors into disentangled embeddings with improved interpretability by encoding polysemous semantics separately. |
| Outcome: | The proposed model can be encoded into multiple sub-embeddings or sub-areas and generates more efficient and effective features for natural language processing. |