ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021.acl-long)
Copied to clipboard
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou
| Challenge: | Existing pre-training objectives do not explicitly model relational facts in text . Experimental results show that ERICA can improve typical PLMs on several language understanding tasks, including relation extraction, entity typing and question answering. |
| Approach: | They propose a contrastive learning framework ERICA to obtain a deep understanding of entities and relations in text. |
| Outcome: | The proposed framework can improve PLMs on several language understanding tasks, especially under low-resource settings. |
Similar Papers
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)
Copied to clipboard
Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu
| Challenge: | Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. |
| Approach: | They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning . |
| Outcome: | The proposed model can encode words into fine-grained representations without modification of production pipelines. |
Contrastive Learning for Task-Independent SpeechLLM-Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in speech processing tasks but their reliance on written text limits their application in real-world scenarios. |
| Approach: | They propose a task-independent speech pretraining stage and task-specific fine-tuning stage to adapt LLMs to speech processing tasks. |
| Outcome: | The proposed model outperforms models specialized on speech translation and question answering while being trained on 10% of the task-specific data. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
CATE: A Contrastive Pre-trained Model for Metaphor Detection with Semi-supervised Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for metaphor detection require a large amount of labeled data and are not linguistically-based. |
| Approach: | They propose a ContrAstive pre-Trained modEl (CATE) for metaphor detection with semi-supervised learning using a pre-trained model to obtain a contextual representation of target words. |
| Outcome: | The proposed model outperforms existing models on several benchmark datasets and achieves better performance against state-of-the-art models. |
Structural Contrastive Pretraining for Cross-Lingual Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train multilingual language models using pretraining tasks like mask language modeling have yielded promising results on a wide range of downstream tasks. |
| Approach: | They propose a new task to align the structural words in a parallel sentence, enhancing models’ ability to comprehend cross-lingual representations. |
| Outcome: | The proposed task improves model's ability to comprehend cross-lingual representations by increasing the frequency of negative pairings. |
Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)
Copied to clipboard
| Challenge: | Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks. |
| Approach: | They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks. |
| Outcome: | The proposed method improves on standard semantic textual similarity and semantic search tasks. |
LaoPLM: Pre-trained Language Models for Lao (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) can capture different levels of concepts in context . previous work on Lao has been hampered by the lack of annotated datasets . |
| Approach: | They construct a text classification dataset to alleviate the resource-scarce situation of Lao . they evaluate them on two downstream tasks: part-of-speech tagging and text classification . |
| Outcome: | The proposed model can capture different levels of concepts in context and generate universal language representations. |
Pre-training Language Models for Comparative Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions. |
| Approach: | They propose a framework to pre-train language models for enhancing comparative reasoning abilities . they collect scalable data for text-based entity comparison . |
| Outcome: | The proposed framework significantly improves comparative reasoning abilities under low-resource conditions on downstream tasks. |
RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual relationship detection models only use numeric ids of relation labels for training, but ignore semantic correlation between labels. |
| Approach: | They propose a visual Relationship prediction framework that transfers natural language knowledge from Contrastive Language-Image Pre-training models to enhance the relationship prediction. |
| Outcome: | The proposed framework improves visual relationship prediction by matching semantic correlations with relation triplets. |