MA-BERT: Learning Representation by Incorporating Multi-Attribute Knowledge in Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for incorporating external attribute knowledge into deep neural networks are concatenating multiple attributes to word/text representation or treating them as biases to adjust attention distribution. |
| Approach: | They propose a multi-attribute BERT to incorporate external attribute knowledge into deep neural networks. |
| Outcome: | The proposed method outperforms existing models and models on three benchmark datasets. |
Similar Papers
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Syntactically Aware Cross-Domain Aspect and Opinion Terms Extraction (2020.coling-main)
Copied to clipboard
| Challenge: | Supervised-learning approaches fail to scale across domains where labeled data is lacking. |
| Approach: | They propose a method for incorporating external linguistic knowledge into a self-attention mechanism coupled with a transformer-based model. |
| Outcome: | The proposed method enables leveraging syntactic knowledge from transformer-based models to bridge the gap between domains. |
Understanding Pre-trained BERT for Aspect-based Sentiment Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show impressive results on aspects-based sentiment analysis tasks. |
| Approach: | They analyze the attentions and learned representations of BERT for aspects-based sentiment analysis tasks. |
| Outcome: | The proposed model can be used for aspects-based sentiment analysis (ABSA) but it is not clear how it can provide important features for downstream tasks. |
BERTGen: Multi-task Generation through BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU). |
| Approach: | They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks. |
| Outcome: | The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts. |
MTRec: Multi-Task Learning over BERT for News Recommendation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing news recommendation methods learn news representations solely based on news titles. Existing methods only utilize title information and neglect other valuable news information such as categories and entities. |
| Approach: | They propose a multi-task method to incorporate multi-field information into BERT, which improves its news encoding capability. |
| Outcome: | Extensive experiments on the MIND news recommendation benchmark show the proposed method is effective. |
Document-Level Neural Machine Translation Using BERT as Context Encoder (2020.aacl-srw)
Copied to clipboard
| Challenge: | Large-scale pre-trained representations such as BERT have been widely used in many natural language understanding tasks. |
| Approach: | They propose to use BERT as a context encoder to achieve document-level contextual information, which is then integrated into both the encoder and decoder. |
| Outcome: | The proposed model outperforms strong document-level machine translation baselines on BLEU score and captures document- level context information to boost translation performance. |
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (D19-55)
Copied to clipboard
| Challenge: | Existing studies on ABSA use a sequence tagging problem to extract aspect-specific opinion words from the sentence given the aspect. |
| Approach: | They build a series of simple yet insightful neural baselines to deal with E2E-ABSA task using contextualized embeddings from pre-trained language models. |
| Outcome: | The proposed architecture outperforms state-of-the-art models even with a simple linear classification layer. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Efficient Learning of Multiple NLP Tasks via Collective Weight Factorization on BERT (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune a model for multiple tasks require a large amount of memory and computing power. |
| Approach: | They propose to factorize the weighs of a pre-trained Transformer model to improve training efficiency across multiple tasks by using BERT-Large as an instantiation of the Transformer and the GLUE as the evaluation benchmark. |
| Outcome: | The proposed method matches or improves the original fine-tuned model’s performance for each task while effectively decreasing parameter requirements by two orders of magnitude. |
Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees (2021.eacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. |
| Approach: | They propose a plug-and-play framework that incorporates syntax trees into pre-trained Transformers. |
| Outcome: | The proposed framework improves on pre-trained models on natural language understanding datasets and shows that it can be used to train pre-structured neural networks. |