More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Latent Dirichlet Allocation models ingest words to discover their latent topics . but it is unclear how to achieve the best results for languages without marked word boundaries . |
| Approach: | They propose to use retokenization to merge frequent token ngrams into collocations in input to a Latent Dirichlet Allocation model. |
| Outcome: | The proposed model improves topic coherence and coherency in Chinese and Thai . the proposed model is more coherent and clearer than unmerged models . |
Similar Papers
Improving Privacy Guarantee and Efficiency of Latent Dirichlet Allocation Model Training Under Differential Privacy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Dirichlet allocation (LDA) is a widely used topic model to discover the latent semantic of text data. |
| Approach: | They propose to combine a subsampling method with CGS to improve efficiency while amplifying privacy by using a novel metric, the efficiency–privacy function. |
| Outcome: | The proposed algorithm improves efficiency while amplifying privacy while subsampling in CGS increases efficiency while preserving privacy. |
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
Latent-Condensed Transformer for Efficient Long Context Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation. |
| Approach: | They propose a Latent-Condensed Attention mechanism that performs structured context condensation directly within MLA's latent space. |
| Outcome: | The proposed approach reduces KV cache size and attention cost without adding parameters. |
Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in Pretrained Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods to adapt pretrained embeddings to a large corpus are limited and do not provide sufficient quality. |
| Approach: | They propose to use a small corpus D_T to pretrain embeddings that accurately capture the sense of words in a limited set of focused topics. |
| Outcome: | The proposed embeddings capture the sense of words in a topic in spite of the limited size of the corpus D_T. |
GraphBTM: Graph Enhanced Autoencoded Variational Inference for Biterm Topic Model (D18-1)
Copied to clipboard
| Challenge: | Conventional topic models suffer different problems in different settings. |
| Approach: | They propose a novel way to model word-pairs named biterms in the whole corpus and a Graph Convolutional Networks (GCNs) with residual connections to extract transitive features from biterm. |
| Outcome: | The proposed model generates more coherent topics compared with previous methods. |
LEPO: Latent Reasoning Policy Optimization for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing latent reasoning methods that use chain of thought (CoT) are limited to selecting one discrete token at each reasoning step, which potentially induces information loss. |
| Approach: | They propose a framework that injects controllable stochasticity into latent reasoning via Gumbel-Softmax, restoring LLMs' exploratory capacity and enhancing their compatibility with Reinforcement Learning (RL). |
| Outcome: | The proposed framework preserves richer information for more comprehensive reasoning and is compatible with Reinforcement Learning (RL). |
Tell, Don’t Show: Leveraging Language Models’ Abstractive Retellings to Model Literary Themes (2025.findings-acl)
Copied to clipboard
| Challenge: | Literature challenges traditional bag-of-words approaches for topic modeling because narrative language focuses on immersive sensory details instead of abstractive description or exposition. |
| Approach: | They propose a topic modeling approach that prompts generative language models to *tell* what passages *show*, thereby translating narratives’ surface forms into higher-level concepts and themes. |
| Outcome: | The proposed model can translate narratives’ surface forms into higher-level concepts and themes than by running LDA alone or directly asking LMs to list topics. |
CAST: Corpus-Aware Self-similarity Enhanced Topic modelling (2025.naacl-long)
Copied to clipboard
Yanan Ma, Chenghao Xiao, Chenhan Yuan, Sabine N Van Der Veer, Lamiece Hassan, Chenghua Lin, Goran Nenadic
| Challenge: | Existing topic modelling methods encode contextual information of documents while ignoring contextual details of candidate centroid words. Existing methods are limited by the contextualization gap. |
| Approach: | They propose a topic modelling method that builds upon candidate centroid word embeddings contextualized on the dataset and a self-similarity-based method to filter out less meaningful tokens. |
| Outcome: | The proposed method significantly enhances the coherence and diversity of generated topics, and handles noisy data, outperforming strong baselines. |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for textual data augmentation (DA) are highly data-hungry and struggle to perform satisfactorily under noisy conditions. |
| Approach: | They propose to leverage a diffusion language model to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens. |
| Outcome: | The proposed method captures in-domain knowledge and generates pseudo samples by reconstructing strong label-related tokens. |