Papers with BERT-base
BERTweet: A pre-trained language model for English Tweets (2020.emnlp-demos)
Copied to clipboard
| Challenge: | Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. |
| Approach: | They propose to train a pre-trained language model for English Tweets using the RoBERTa pre training procedure and use it to train the model. |
| Outcome: | Experiments show that the model outperforms baseline models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. |
Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for question answering are limited in the availability of labeled data. |
| Approach: | They propose a hierarchical conditional variational autoencoder for generating QA pairs given unstructured texts as contexts while maximizing mutual information between generated QA pair to ensure consistency. |
| Outcome: | The proposed framework achieves impressive performance gains over baseline models on both tasks, using only a fraction of data for training. |
Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension? (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks for machine reading comprehension (MRC) are insufficient to assess models for their capabilities to read and comprehend . |
| Approach: | They propose an ablation-based method to assess the extent to which MRC datasets evaluate the understanding of explicit discourse relations. |
| Outcome: | The proposed method shows that the model's performance drops on three large-scale datasets . the results suggest that most of the answers do not require understanding the discourse structure of the text. |
PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work often tackles the problem of text classification when there is a limited amount of training data. |
| Approach: | They propose a method to generate more helpful augmented data by utilizing the LLM's ability to follow instructions and perform few-shot classifications. |
| Outcome: | The proposed method generates more helpful examples near class boundaries, but generating borderline examples increases the risk of false positives in the dataset. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that learning multiple training objectives in a single model makes the learned language representation barely converge to the desired optimum. |
| Approach: | They propose a meta-learning-based adaptive sampler which learns latent sampling pattern on arbitrary pre-training objectives. |
| Outcome: | Empirical studies show that learning multiple objectives in a single model makes it difficult to achieve the desired optimum. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |
BERT for Coreference Resolution: Baselines and Analysis (D19-1)
Copied to clipboard
| Challenge: | Recent BERT-based models have reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition. |
| Approach: | They apply BERT to coreference resolution, achieving a new state of the art on the GAP and OntoNotes benchmarks. |
| Outcome: | A qualitative analysis of model predictions shows that BERT-large is better at distinguishing between related but distinct entities, but there is room for improvement in modeling document-level context, conversations, and mention paraphrasing. |
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune deep pretrained language models face catastrophic forgetting problems. |
| Approach: | They propose a recall and learn mechanism which integrates pretraining and downstream tasks into a single mechanism. |
| Outcome: | The proposed method achieves state-of-the-art performance on the GLUE benchmark and better average performance than directly fine-tuning of BERT-large. |
Rank-Awareness and Angular Constraints: A New Perspective on Learning Sentence Embeddings from NLI Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | High-quality sentence embeddings are critical for advancing a wide range of Natural Language Processing tasks. |
| Approach: | They propose a framework that leverages the full NLI dataset augmented with pre-computed continuous similarity scores (S) they employ a Rank Margin objective that enforces rank consistency against S using an explicit margin and a Gated Angular objective that conditionally refines embedding geometry based on NLI label (L) and S score agreement. |
| Outcome: | The proposed framework outperforms baseline models on STS and the MTEB benchmarks. |