Modeling Text using the Continuous Space Topic Model with Pre-Trained Word Embeddings (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing topic models that extract latent topics from text are based on latent topic and do not use intermediate variables such as latent subjects. |
| Approach: | They propose a model that extends the continuous space topic model (CSTM) they pre-train word embeddings which capture the semantics of words and plug them into the CSTM . |
| Outcome: | The proposed model performs better than the baseline model in terms of perplexity and convergence speed. |
Similar Papers
Dynamic Topic Modeling by Clustering Embeddings from Pretrained Language Models: A Research Proposal (2022.aacl-srw)
Copied to clipboard
| Challenge: | Neural Topic Models (NTMs) are topic models that are created with the help of a pretrained language model. |
| Approach: | They propose to do Neural Topic Modeling by Clustering document Embeddings (NTM-CE) with a pretrained language model to create dynamic topic models. |
| Outcome: | The proposed model can be evaluated theoretically and practically using quantitative measurements of coherence and human evaluation to evaluate the model. |
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models rely on probabilistic models to uncover themes within document collections, but are they the only option? |
| Approach: | They propose a way to cluster pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. |
| Outcome: | The proposed approach performs as well as classical topic models, but with lower runtime and computational complexity. |
Multi-source Neural Topic Modeling in Multi-view Embedding Spaces (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work has used pre-trained word embeddings to address data sparsity in short-text or small document collections. |
| Approach: | They propose a neural topic modeling framework using multi-view embedding spaces to improve topic quality and deal with polysemy. |
| Outcome: | The proposed framework improves topic quality and deal with polysemy. |
Topic Modeling in Embedding Spaces (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. |
| Approach: | They propose an embedded topic model that integrates word embeddings with a categorical distribution that is the natural parameter between the word’s embeddment and an embeddement of its assigned topic. |
| Outcome: | The embedded topic model outperforms existing topic models in terms of topic quality and predictive performance. |
CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing topic models rely on bag-of-words (BOW) representations to capture word order information. |
| Approach: | They propose a neural topic model that integrates contextualized word embeddings from BERT to learn the topic vector of a document without BOW information. |
| Outcome: | The proposed model generates more coherent and meaningful topics compared to existing models while accommodating unseen words in newly encountered documents. |
Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that using external knowledge such as pre-trained word embeddings or pre-train language models only achieved limited performance improvements but with huge computational overhead. |
| Approach: | They propose to incorporate external knowledge into neural topic modeling by pre-trained word embeddings (PWEs) or pre-train language models (PLMs) they propose to fine-tune the neural topic model on the target dataset and reduce the huge size of training data. |
| Outcome: | The proposed approach outperforms current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs on three datasets and greatly reduces the huge size of training data. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2021.acl-short)
Copied to clipboard
| Challenge: | Recent neural topic models extract words from documents, but they are not coherent . coherence is crucial for topic models, but many use bag-of-words document representations as input . pre-trained language models are becoming ubiquitous in natural language processing . |
| Approach: | They combine contextualized representations with neural topic models to produce more coherent topics . they say that future improvements in language models will translate into better topic models . |
| Outcome: | The proposed approach produces more meaningful and coherent topics than bag-of-words models and recent neural models. |
The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation (2024.naacl-short)
Copied to clipboard
| Challenge: | Continuous-output neural machine translation models are trained to predict the continuous representation based on distances between vectors. |
| Approach: | They propose a continuous-output neural machine translation (CoNMT) approach that uses random output embeddings to outperform laboriously pre-trained models. |
| Outcome: | The proposed strategy outperforms pre-trained embeddings on large datasets and is strongest for rare words due to the geometry of their embedders. |