How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection (N19-1)
Copied to clipboard
| Challenge: | Using a pre-defined vocabulary is a common approach to selecting text inputs . however, using a large vocabulary is not economical, as it limits the model's applicability on computation-or memoryconstrained scenarios. |
| Approach: | They propose a more sophisticated variational vocabulary dropout to perform vocabulary selection . they propose two new metrics to measure area under accuracy-vocab curve and Vocab Size under X% accuracy drop . |
| Outcome: | The proposed framework outperforms the baselines on the vocabulary selection problem on multiple NLP classification tasks. |
Similar Papers
Large Vocabulary Size Improves Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have investigated the properties of internal layers in large language models, but no studies have defined the vocabulary size. |
| Approach: | They propose a method to use a new vocabulary instead of the pre-defined one in a continual training scenario. |
| Outcome: | The proposed method outperforms the model with the pre-defined vocabulary in a continual training scenario. |
The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural Machine Translation models can be optimized to improve latency by constraining the set of output words . lexical shortlisting fails to select the right set of input words for semantically non-compositional phenomena such as idiomatic expressions. |
| Approach: | They propose a model of vocabulary selection that constrains the set of allowed output words . they propose to increase the size of the allowed set to restore translation quality . |
| Outcome: | The proposed model restores translation quality of an unconstrained system, as measured by human evaluations on WMT newstest2020 and idiomatic expressions, at an inference latency competitive with alignment-based selection using aggressive thresholds. |
Finding the Optimal Vocabulary Size for Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Class imbalance is said to exist when one or more classes are not of approximately equal frequency in data. |
| Approach: | They cast neural machine translation (NMT) as a classification task in an autoregressive setting and examine its limitations. |
| Outcome: | The proposed model performs better on multiple languages with large data sizes with different vocabulary sizes. |
VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing static vocabulary pruning designs that reduce memory usage suffer from rigid, one-size-fits-all designs that cause information loss during the prefill stage and lack flexibility. |
| Approach: | They propose a decoupled dynamic vocabulary selection framework that addresses memory constraints through offloading embedding and implements a hybrid static-dynamic vocabulary selection strategy for LM Head. |
| Outcome: | The proposed framework reduces memory usage by 99% with minimal or no degradation in performance. |
Rethinking Vocabulary Augmentation: Addressing the Challenges of Low-Resource Languages in Multilingual Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to augment vocabularies ignore the disparities between model representation and frequency distributions. |
| Approach: | They propose an Entropy-Consistency Word Selection method which integrates semantic and frequency metrics for vocabulary augmentation. |
| Outcome: | The proposed method improves performance for low-resource languages compared to high-resourced ones . it integrates semantic and frequency metrics for vocabulary augmentation . |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
Vocabulary Learning via Optimal Transport for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that VOLT beats widely-used vocabularies in diverse scenarios, including WMT-14 English-German translation, TED bilingual translation, and TED multilingual translation. |
| Approach: | They propose a token dictionary solution that can be used without trial training to find the best dictionary with a proper size. |
| Outcome: | The proposed solution beats widely-used vocabularies in English-German translation, TED bilingual translation, and TED multilingual translation. |
Improving Lexical Choice in Neural Machine Translation (N18-1)
Copied to clipboard
| Challenge: | False positives: the output layer rewards frequent words disproportionately, we argue . Falsibles: a model that learns word representations in continuous space tends to translate rare words . |
| Approach: | They propose to fix the norms of both vectors to a constant value and integrate a lexical module which is jointly trained with the rest of the model. |
| Outcome: | The proposed approach achieves improvements of up to +4.3 BLEU surpassing phrase-based translation in nearly all settings. |
Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification (2021.naacl-main)
Copied to clipboard
| Challenge: | Many text classification algorithms depend on the size of the corpus’ vocabulary due to their bag-of-words representation. |
| Approach: | They propose to evaluate how preprocessing techniques affect the run-time of models by evaluating ten techniques over four models and two datasets. |
| Outcome: | The proposed methods can reduce run-time with no loss of accuracy while sacrificing up to 65%. |
Efficient Vocabulary Reduction for Small Language Models (2025.coling-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have high computational costs and energy consumption, making their deployment in industrial settings difficult. |
| Approach: | They propose a small language model that compresses the embedding layer and reduces model size without significant loss of performance. |
| Outcome: | The proposed model reduces the embedding layer while maintaining performance while improving accuracy and performance. |