Disentangling Transformer Language Models as Superposed Topic Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Topic Modelling is an established area of research where the quality of a given topic is measured using coherence metrics. |
| Approach: | They propose a weight-based approach to search and disentangle decoder-only TLM by a Wikipedia corpus. |
| Outcome: | The proposed approach can be applied to GPT-2 models and LLaMA models. |
Similar Papers
Towards Reinterpreting Neural Topic Models via Composite Activations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Most Neural Topic Models (NTMs) use a variational auto-encoder framework producing K topics limited to the size of the encoder’s output. |
| Approach: | They propose a model-free two-stage process to reinterpret NTM and derive further insights on the state of the trained model. |
| Outcome: | The proposed model-free process decouples the strict interpretation of topics from the original NTM and evaluates them on a large external corpus. |
Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work incorporates pre-trained word embeddings into Neural Topic Models (NTMs), generating highly coherent topics. |
| Approach: | They conduct thorough experiments to investigate whether embeddings directly with an appropriate word selection method can generate more coherent and diverse topics than NTMs. |
| Outcome: | The proposed model generates more coherent and diverse topics than traditional NTMs, achieving higher efficiency and simplicity. |
Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for text segmentation use supervised and unsupervised learning to perform tasks such as text summarization and keyword extraction. |
| Approach: | They propose a transformer over transformer framework to perform neural text segmentation. |
| Outcome: | The proposed framework outperforms state-of-the-art models in terms of semantic coherence measure . bottom-level sentence encoders pre-trained on specific languages yield better performance . |
DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Topic Models and Large Language Models (LLMs) primarily use contextual embeddings from LLMs, which are not optimal for clustering or topic generation. |
| Approach: | They propose a framework that leverages Encoder-Decoders to generate highly clusterable embeddings that could generate topics that exhibit enhanced clusterability and enhanced semantic coherence compared to existing methods. |
| Outcome: | The proposed framework is efficient to train and exhibits high adaptability, demonstrating its potential for a wide array of applications. |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
Self-Supervised Neural Topic Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Topic models are useful tools for analyzing and interpreting the main underlying themes of large corpora of text. |
| Approach: | They propose a self-supervised neural topic model that learns a topic representation jointly from three co-occurring words and a document that the triple originates from. |
| Outcome: | The proposed model outperforms existing topic models in coherence metrics and document clustering accuracy. |
Unified Neural Topic Model via Contrastive Learning and Term Weighting (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent techniques employ pretrained language models to improve topic quality. |
| Approach: | They propose a topic-based model that uses contrastive learning and term weighting to learn from a pretrained language model and discover influential terms from semantically coherent clusters. |
| Outcome: | The proposed model outperforms baselines across multiple topic coherence measures and can be used as an add-on to existing topic models and improves their performance. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
Neural Topic Modeling with Large Language Models in the Loop (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated promising capabilities in topic discovery, but their direct application to topic modeling suffers from issues such as incomplete topic coverage, misalignment of topics, and inefficiency. |
| Approach: | They propose a novel LLM-in-the-loop framework that integrates Large Language Models with Neural Topic Models (NTMs) global topics and document representations are learned through the NTM, while an LLM refines these topics using an Optimal Transport (OT)-based alignment objective. |
| Outcome: | The proposed framework improves topic interpretability while preserving the efficiency of existing NTMs. |
Enriching and Controlling Global Semantics for Text Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models have been proven effective in creating fluent and informative summaries, but they suffer from the short-range dependency problem, causing them to produce summary that miss the key points of document. |
| Approach: | They propose a neural topic model empowered with normalizing flow to capture global semantics of the document and integrate them into the summarization model. |
| Outcome: | The proposed model outperforms state-of-the-art summarization models on five common text summarizing datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and PubMed. |