FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on word embeddings and pre-trained language models has shown the large impact of language representations on natural language processing (NLP) models across tasks and domains. |
| Approach: | They propose feature-based adversarial meta-embeddings with an attention function that is guided by word-specific properties, such as shape and frequency, to handle subword-based embeddings. |
| Outcome: | The proposed model improves performance in downstream tasks even with word embeddings from transformers. |
Similar Papers
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained language models exhibit poor generalization and robustness in adversarial settings. |
| Approach: | They propose a self-supervised sentence embedding framework that improves generalization and robustness against adversarial attacks. |
| Outcome: | The proposed framework reduces the success rate of adversarial attacks by almost half . it also improves semantic text similarity tasks and various transfer tasks . |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
Task-oriented Domain-specific Meta-Embedding for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods neglect domain-specific knowledge and use the same word embedding for each word in all domain-specified datasets. |
| Approach: | They propose a method to incorporate domain-specific and task-oriented information into meta-embeddings by combining pre-trained word embeddings. |
| Outcome: | The proposed method performs well on four text classification datasets and shows that it is compatible with existing methods. |
Shallow Domain Adaptive Embeddings for Sentiment Analysis (D19-1)
Copied to clipboard
| Challenge: | Existing domain adaptation algorithms for text classification are limited by lack of training data and exploiting domain idiosyncrasies to improve performance. |
| Approach: | They propose a domain adaptation layer that learns weights to combine a generic and a specific word embedding into a DA embeddable. |
| Outcome: | The proposed approach improves on binary and multi-class classification tasks using popular encoder architectures. |
Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings (N18-2)
Copied to clipboard
| Challenge: | Existing methods for producing word embeddings have shown to produce accurate meta-embeddings from pre-trained source embeddables. |
| Approach: | They propose to use arithmetic mean of two distinct word embedding sets to produce an accurate meta-embedding. |
| Outcome: | The proposed method produces meta-embeddings comparable or better than more complex methods. |
GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer-based architectures provide impressive results in many NLP tasks, but obtaining high-quality annotated data is expensive and time consuming. |
| Approach: | They propose a semisupervised learning method that ex- tends the fine-tuning of BERT-like architectures with unlabeled data in a generative adversarial setting. |
| Outcome: | The proposed method reduces the requirement for annotated examples while achieving good performance in sentence classification tasks. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
Robust to Noise Models in Natural Language Processing Tasks (P19-2)
Copied to clipboard
| Challenge: | Existing spelling correction systems are far from perfect for noise-sensitive texts . a new way to handle noise is to make models robust to noise. |
| Approach: | They propose a robust to noise word embeddings model which outperforms existing models in different tasks. |
| Outcome: | The proposed model outperforms existing models in three downstream tasks and shows improvements in noise robustness over existing models. |
RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing PLMs suffer from poor robustness in adversarial scenarios, despite their success with unseen samples. |
| Approach: | They propose a self-supervised sentence embedding framework that enhances generalization and robustness in various text representation tasks and against diverse adversarial attacks. |
| Outcome: | The proposed framework improves generalization and robustness in various representation tasks and against diverse adversarial attacks. |