| Challenge: | Current paradigms for transfer learning use general knowledge as a foundation for more specialized endeavors. |
| Approach: | They propose to combine probabilistic topic models and pretrained transformers to improve topic quality by using knowledge distillation. |
| Outcome: | The proposed framework improves topic quality over all estimated topics and in head-to-head comparisons of aligned topics. |
Similar Papers
Coherence-Aware Neural Topic Modeling (D18-1)
Copied to clipboard
| Challenge: | Topic models are evaluated for their ability to describe documents well (i.e. low perplexity) topic coherence is not optimized for and is only evaluated after training. |
| Approach: | They propose to incorporate a topic coherence objective into the training process by incorporating a coherency objective into a model. |
| Outcome: | The proposed model exhibits similar level of perplexity as baseline models but significantly higher topic coherence. |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that using external knowledge such as pre-trained word embeddings or pre-train language models only achieved limited performance improvements but with huge computational overhead. |
| Approach: | They propose to incorporate external knowledge into neural topic modeling by pre-trained word embeddings (PWEs) or pre-train language models (PLMs) they propose to fine-tune the neural topic model on the target dataset and reduce the huge size of training data. |
| Outcome: | The proposed approach outperforms current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs on three datasets and greatly reduces the huge size of training data. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Are Neural Topic Models Broken? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation paradigms are often divorced from real-world use . recent results have challenged the validity of the prevailing model evaluation paradigm . |
| Approach: | They show that neural topic models fare worse in both respects compared to an established classical method. |
| Outcome: | The proposed method outperforms the members of the ensemble in both respects. |
Benchmarking Neural Topic Models: An Empirical Study (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural topic modeling has been attracting much attention recently due to its ability to leverage the advantages of both neural networks and probabilistic topic models. |
| Approach: | They propose to evaluate neural topic models in three tasks using large datasets and a set of metrics to compare them. |
| Outcome: | The proposed models perform better in the first and third tasks than the traditional probabilistic models and are better in many cases. |
Neural Topic Modeling via Contextual and Graph Information Fusion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models generate uninformative and incoherent topics that hinder interpretable insights from managing textual data. |
| Approach: | They propose to incorporate contextual and graph information to improve the variational autoencoder framework by combining contextual and bag-of-words information. |
| Outcome: | The proposed framework generates more coherent and diverse topics on three benchmark datasets and achieves strong performance on automatic and manual evaluations. |
Diversity-Aware Coherence Loss for Improving Neural Topic Models (2023.acl-short)
Copied to clipboard
| Challenge: | Experimental results show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters. |
| Approach: | They propose a variational autoencoder framework that minimizes the posterior and prior divergence and a diversity-aware coherence loss that encourages the model to learn corpus-level coherency scores while maintaining high diversity between topics. |
| Outcome: | The proposed approach significantly improves the performance of neural topic models without pretraining or additional parameters. |
HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in neural topic models (NTMs) have improved topic quality but still face challenges: weak document-topic alignment, high inference costs due to large pretrained language models, and limited modeling of hierarchical topic structures. |
| Approach: | They propose a framework that integrates hierarchical clustering and contrastive learning to refine document-topic relationships using compact PLM-based embeddings. |
| Outcome: | The proposed framework improves topic coherence, topic performance, representation quality and computational efficiency over existing NTMs. |
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |