PUNR: Pre-training with User Behavior Modeling for News Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing news recommendation methods use pre-trained language models to produce news vectors and user vectors. |
| Approach: | They propose an unsupervised pre-training paradigm with two tasks for user behavior modeling. |
| Outcome: | The proposed model improves on the real-world news benchmark. |
Similar Papers
Neural News Recommendation with Heterogeneous User Behavior (D19-1)
Copied to clipboard
| Challenge: | Existing news recommendation methods rely on news click history to model user interest, but data sparsity is a problem . other kinds of user behaviors such as webpage browsing and search queries can provide useful clues of users’ news reading interest. |
| Approach: | They propose to exploit heterogeneous user behaviors to learn news representations from their titles via CNN networks and apply attention networks to select important words. |
| Outcome: | The proposed approach exploits heterogeneous user behaviors on a real-world dataset. |
PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for user modeling cannot exploit useful information in unlabeled data . Existing models only model task-specific user information and do not exploit universal user information encoded in user behaviors. |
| Approach: | They propose to pre-train user models from large-scale unlabeled user behavior data. |
| Outcome: | The proposed method can model relatedness between historical and future behaviors on two real-world datasets. |
Neural News Recommendation with Long- and Short-term User Representations (P19-1)
Copied to clipboard
| Challenge: | Existing news recommendation methods learn a single representation for each user, which may be insufficient. |
| Approach: | They propose a neural news recommendation approach which can learn both long- and short-term user representations by using a news encoder and a user encoder. |
| Outcome: | The proposed approach can learn both long- and short-term user representations on a real-world dataset. |
Boosting Deep CTR Prediction with a Plug-and-Play Pre-trainer for News Recommendation (2022.coling-1)
Copied to clipboard
| Challenge: | Personalized news recommendation is a ubiquitous channel in various online applications, such as Google News and MSN News. |
| Approach: | They propose a plug-and-play pre-trainer to learn both user and news encoders through multi-task pre-training. |
| Outcome: | The proposed model improves on existing models and improves inference and updating time. |
Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages. |
| Approach: | They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences . |
| Outcome: | The proposed method can be used on four sentence pair modeling tasks in English and Japanese. |
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance . |
| Approach: | They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible . |
| Outcome: | The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes . |
An Empirical Exploration of Local Ordering Pre-training for Structured Prediction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-training contextualized encoders with language model objectives is effective for structured prediction. |
| Approach: | They propose a semi-supervised method for pre-training contextualized encoders with language model objectives. |
| Outcome: | The proposed method is effective on three typical structured prediction tasks in four languages. |
Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work fine tunes the PLM with the news recommendation task, which can cause a domain shift problem. |
| Approach: | They propose a self-supervised method to adapt general PLM to news domain with a contrastive matching task between news titles and news bodies. |
| Outcome: | The proposed method can improve both the effectiveness and efficiency of the large PLM-based news recommendation model while maintaining its performance. |
Train No Evil: Selective Masking for Task-Guided Pre-Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models can't capture domain-specific and task-specific patterns because of the task-agnostic pre-training stage. |
| Approach: | They propose a task-guided pre-training stage with selective masking between general pre-train and fine-tuning to learn domain-specific patterns. |
| Outcome: | The proposed method can achieve comparable or even better performance with less than 50% of computation cost. |
Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are limited . we propose a few-shot fine-uning framework for NER . |
| Approach: | They propose a few-shot fine-tuning framework for named entity recognition (NER) they propose three new types of tokens, "is-entity", "which-type" and "bracket" |
| Outcome: | The proposed framework improves on pre-trained language models on several benchmark datasets. |