| Challenge: | Byte-pair encoding (BPE) is a popular method of tokenizing valid words onto a token space V b with a predetermined fixed size, and handling out-of-vocabulary words, breaking words into smaller tokens. |
| Approach: | They propose to interpret the recovery of valid words from these tokens as a ranking problem and apply existing evaluation measures to topic sets. |
| Outcome: | The proposed model interprets the recovery of valid words from these tokens as a ranking problem and applies existing evaluation measures. |
Similar Papers
Investigating the Effectiveness of BPE: The Power of Shorter Sequences (D19-1)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, but its reasons for its effectiveness are not well understood. |
| Approach: | They link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family. |
| Outcome: | The proposed method is compared with dictionary-based compression algorithms and improves on a fixed vocabulary size budget. |
Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for topic model evaluation use automated measures modeled on human evaluation tests that are dissimilar to applied usage. |
| Approach: | They propose to use a novel experimental framework to evaluate topic models and assess their coherence for specialized collections in an applied setting. |
| Outcome: | The proposed framework is reflective of human evaluations using open labeling, typical of applied research. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Topic Modeling: Contextual Token Embeddings Are All You Need (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current neural approaches to topic modeling have not been able to solve all of the problems. |
| Approach: | They propose a topic modeling approach that uses document contextual token embeddings to find topics and find topic spans within documents. |
| Outcome: | The proposed model outperforms the current state-of-the-art models on a comprehensive set of topic model evaluation metrics. |
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization (2026.acl-long)
Copied to clipboard
Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich
| Challenge: | Tokenization is the first step of most NLP pipelines. |
| Approach: | They propose a parity-aware byte pair encoder that maximizes the compression gain of the currently worst-compressed language for cross-lingual parity. |
| Outcome: | a new algorithm reduces tokenization inequality by 89% compared to classical BPE . the proposed algorithm is based on a fair-max rule that maximizes the compression gain of the currently worst-compressed language . |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current vocabulary adaptation approaches append the target domainspecific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Approach: | They propose a vocabulary adaptation scheme that appends a target domain-specific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Outcome: | The proposed approach improves by 3.57% (in terms of accuracy) and 1.87% (royal-L) over various classification and summarization tasks. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |
A Bayesian Topic Model for Human-Evaluated Interpretability (2022.lrec-1)
Copied to clipboard
| Challenge: | Topic modeling is an effective way to analyze unstructured textual data. |
| Approach: | They propose to combine nonparametric and weakly-supervised topic models to produce interpretable topics. |
| Outcome: | The proposed model outperforms weakly-supervised models in the field of topic modeling. |
Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work shows that byte-pair encoding (BPE) tokenization uses a learned merge list to iteratively combine subword units into tokens during inference time. |
| Approach: | They propose to use a standard byte-pair encoding algorithm to pair a learned token vocabulary with a detailed merge list to compress text. |
| Outcome: | The proposed algorithms differ from the encoding process during training and show that the targetted deviation from merge lists exhibits significant degradation in language model performance. |