| Challenge: | Topic coherence is increasingly being used to evaluate topic models and filter topics for end-user applications. |
| Approach: | They propose to use topic intrusion to guess an outlier topic given a document and a few topics to automate the task. |
| Outcome: | The proposed method improves upon the state-of-the-art method and shows it can be used as an alternative to topic perplexity evaluation. |
Similar Papers
Coherence-Aware Neural Topic Modeling (D18-1)
Copied to clipboard
| Challenge: | Topic models are evaluated for their ability to describe documents well (i.e. low perplexity) topic coherence is not optimized for and is only evaluated after training. |
| Approach: | They propose to incorporate a topic coherence objective into the training process by incorporating a coherency objective into a model. |
| Outcome: | The proposed model exhibits similar level of perplexity as baseline models but significantly higher topic coherence. |
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis (2024.eacl-long)
Copied to clipboard
Zongxia Li, Andrew Mao, Daniel Stephens, Pranav Goel, Emily Walpole, Alden Dima, Juan Fung, Jordan Boyd-Graber
| Challenge: | Existing evaluation metrics such as coherence and coherency are inadequate for neural topic models. |
| Approach: | They conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. |
| Outcome: | The proposed model performs better on cluster evaluation metrics and human evaluations than classical models on real-world tasks. |
Revisiting Automated Topic Model Evaluation with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic models are an unsupervised dimensionality reduction technique that help organize large text collections. |
| Approach: | They propose to use large language models to evaluate document output and determine optimal number of topics. |
| Outcome: | The proposed model performs better on coherence ratings of word sets than on intrustion detection. |
Are Neural Topic Models Broken? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation paradigms are often divorced from real-world use . recent results have challenged the validity of the prevailing model evaluation paradigm . |
| Approach: | They show that neural topic models fare worse in both respects compared to an established classical method. |
| Outcome: | The proposed method outperforms the members of the ensemble in both respects. |
Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for topic model evaluation use automated measures modeled on human evaluation tests that are dissimilar to applied usage. |
| Approach: | They propose to use a novel experimental framework to evaluate topic models and assess their coherence for specialized collections in an applied setting. |
| Outcome: | The proposed framework is reflective of human evaluations using open labeling, typical of applied research. |
Topic Modeling: Contextual Token Embeddings Are All You Need (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current neural approaches to topic modeling have not been able to solve all of the problems. |
| Approach: | They propose a topic modeling approach that uses document contextual token embeddings to find topics and find topic spans within documents. |
| Outcome: | The proposed model outperforms the current state-of-the-art models on a comprehensive set of topic model evaluation metrics. |
Contextualized Topic Coherence Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing topic models that estimate the interpretability of topics are difficult to compare due to their nature as unsupervised models. |
| Approach: | They propose to use contextualized topic coherence metrics to simulate human-centered coherency evaluation while maintaining the efficiency of other automated methods. |
| Outcome: | The proposed metrics better reflect human judgment on topics extracted from short text collections by avoiding highly scored topics that are meaningless to humans. |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
Benchmarking Neural Topic Models: An Empirical Study (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural topic modeling has been attracting much attention recently due to its ability to leverage the advantages of both neural networks and probabilistic topic models. |
| Approach: | They propose to evaluate neural topic models in three tasks using large datasets and a set of metrics to compare them. |
| Outcome: | The proposed models perform better in the first and third tasks than the traditional probabilistic models and are better in many cases. |
A Bayesian Topic Model for Human-Evaluated Interpretability (2022.lrec-1)
Copied to clipboard
| Challenge: | Topic modeling is an effective way to analyze unstructured textual data. |
| Approach: | They propose to combine nonparametric and weakly-supervised topic models to produce interpretable topics. |
| Outcome: | The proposed model outperforms weakly-supervised models in the field of topic modeling. |