Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for solving common NLP tasks rely on fine-tuning of pre-trained transformer models. |
| Approach: | They propose a scoring method that casts a plausibility ranking task in full-text format without fine-tuning . they use masked language modeling head tuned during pre-training phase to exploit this method . |
| Outcome: | The proposed method produces strong baselines comparable to supervised approaches. |
Similar Papers
Bridging the Gap between Pre-Training and Fine-Tuning for Commonsense Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods focusing on this task usually concatenate the concatened concepts words as the inputs of a pre-trained language model (PLM) however, in pre-training, the input is often corrupted sentences with correct word order. |
| Approach: | They propose a two-stage framework to improve the ability of pre-trained language models to deal with masked sentences with incorrect word order and a special token to make the input distribution more similar to the one used in pre-training. |
| Outcome: | The proposed method is able to generate a sentence containing all given concepts and correctly describe the relations between concepts. |
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes lightweight updates to improve commonsense reasoning models . fine-tuning can cause models to overfit to task-specific data and forget knowledge gained during training . |
| Approach: | They propose to use lightweight models to update pre-trained language models to learn commonsense background knowledge. |
| Outcome: | The proposed models learn from commonsense reasoning datasets, but they are overfitted and limited generalized. |
Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to generative commonsense reasoning hypothesize that pre-trained models lack sufficient parametric knowledge for this task. |
| Approach: | They propose to use order-agnostic input to elaborately manipulate the order of the given concepts before generation to evaluate their commonsense knowledge. |
| Outcome: | The proposed approach outperforms more sophisticated models with a lot of external data and resources in the task of generating a logical sentence from a set of concepts. |
Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training a language model by self-supervised tasks on huge datasets and fine-tuning with small labelled data are often inadequate for scientific NER tasks. |
| Approach: | They propose to introduce a "pre-fine tuning" step between pre-training and fine-tuning to construct a corpus by selecting sentences from unlabeled documents that are the most relevant with labelled training data. |
| Outcome: | The proposed approach improves on seven benchmarks on the performance of the proposed model on labelled datasets. |
Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference (2023.acl-long)
Copied to clipboard
Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen
| Challenge: | Existing studies attribute catastrophic forgetting to fine-tuning, and they retain pre-trained knowledge indiscriminately without identifying what knowledge is transferable. |
| Approach: | They propose a unified objective for fine-tuning to retrieve the causality back from pre-trained data and use it to mitigate negative transfer while preserving knowledge. |
| Outcome: | The proposed method outperforms state-of-the-art fine-tuning methods on commonsense QA datasets and can be implemented as a plug-in module to inflate the performance of existing QA models. |
Commonsense Knowledge Transfer for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have transformed the landscape of natural language processing. |
| Approach: | They propose a framework to transfer commonsense knowledge stored in a neural commonsensing model to a general-purpose pre-trained language model. |
| Outcome: | Empirical results show that the proposed framework improves the model’s performance on downstream tasks that require commonsense reasoning. |
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques to fine-tune pre-trained language models on downstream tasks are inadequate. |
| Approach: | They propose a technique to perturb hidden Transformers representations by enhancing generalization of hidden representations from different layers. |
| Outcome: | The proposed technique outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks. |
| Approach: | They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity. |
| Outcome: | The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks. |
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning methods require additional measures after pre-training and before fine-uning. |
| Approach: | They propose to take parameter-efficient fine-tuning into consideration after pre-training and before fine-uning and use meta-learning to prime a model specifically for parameter-efficiency. |
| Outcome: | The proposed method improves on a pre-trained model with certain modifications and achieves 4.96 points on cross-lingual NER fine-tuning. |