Papers with GLUE
LV-BERT: Exploiting Layer Variety for BERT (2021.findings-acl)
Copied to clipboard
| Challenge: | Modern pre-trained language models are mostly built upon stereotyped development sets . LV-BERT model obtained by our method outperforms BERT on various downstream tasks . |
| Approach: | They propose to exploit layer variety from the layer type set and the layer order to improve pre-trained models. |
| Outcome: | The proposed model outperforms BERT and its variants on various downstream tasks. |
Weight-Inherited Distillation for Task-Agnostic BERT Compression (2024.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a predominant approach for BERT compression. |
| Approach: | They propose a weight-inherited distillation method which directly transfers knowledge from the teacher to a compact student model by inheriting the weights. |
| Outcome: | The proposed method outperforms state-of-the-art KD-based methods on GLUE and SQUAD benchmarks. |
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models (2024.acl-long)
Copied to clipboard
Haoran Li, Dadi Guo, Donghao Li, Wei Fan, Qi Hu, Xin Liu, Chunkit Chan, Duanyi Yao, Yuan Yao, Yangqiu Song
| Challenge: | generative large language models (LLMs) exhibit surprising capability and integrate previous tasks into a unified text generation formulation. |
| Approach: | They propose a privacy evaluation benchmark to quantify the privacy leakage of language models. |
| Outcome: | The proposed benchmark compares PPLMs with different privacy implementations to find out how privacy leakage is handled. |
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Mixer encoder model outperforms BERT on GLUE and SuperGLUE, trains 65% faster and runs inference 61% faster. |
| Approach: | They combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. |
| Outcome: | The proposed model outperforms BERT on GLUE and SuperGLUE but trains and runs twice as fast. |
Pretraining Without Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that state-space models (SSMs) outperform standard and deep learning for long-range sequence modeling. |
| Approach: | They propose a model that combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. |
| Outcome: | The proposed model outperforms standard and standard sequence modeling architectures on speech generation and the long range arena benchmarks. |
Efficient Transformer Knowledge Distillation: A Performance Review (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Pretrained transformer language models have been gaining popularity in the field of natural language processing . however, there is no study into the intersection of these two fields . |
| Approach: | They propose a method to extract knowledge from transformers to produce high-performing efficient attention models with low costs. |
| Outcome: | The proposed model compression method preserves up to 98.6% of original model performance across short-context tasks and up to 95.8% on long-concept Named Entity Recognition tasks while decreasing inference times by up to 57%. |
Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)
Copied to clipboard
Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang
| Challenge: | Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks. |
| Approach: | They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator. |
| Outcome: | The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings. |
Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from BERT into a model with large parameters are not efficient due to their large-scale and high computational cost. |
| Approach: | They propose a sentence representation approximating oriented distillation framework that can distill pre-trained BERT into a simple LSTM based model without specifying tasks. |
| Outcome: | The proposed model outperforms other distillation methods and larger models on multiple NLP tasks with efficiency well-improved. |
Layerwise universal adversarial attack on NLP models (2023.findings-acl)
Copied to clipboard
| Challenge: | a new technique for layerwise UATs searches hidden layers of a network for universal adversarial triggers . a previous study showed that adversarials can fool models by perturbing samples that leave the ground truth label unchanged but can modify model prediction drastically. |
| Approach: | They propose a new approach to construct layerwise UATs by perturbing hidden layers of a network and propose LUATs that are more efficient than vanilla UAT methods. |
| Outcome: | The proposed method provides better transferability in a model-to-model setting with an average gain of 9.3% in fooling rate over baseline. |
Shuffled-token Detection for Refining Pre-trained RoBERTa (2021.naacl-srw)
Copied to clipboard
| Challenge: | State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks. |
| Approach: | They propose to refine a pre-trained NLP model by detecting shuffled tokens . they use a sequential approach to train a model using random shuffling . |
| Outcome: | The proposed model achieves better performance on 4 out of 7 GLUE tasks. |
Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders (2025.coling-industry)
Copied to clipboard
| Challenge: | Large-scale e-commerce search systems typically follow a multi-step process to retrieve relevant products for a given query. |
| Approach: | They propose a distillation approach that uses "rationales" generated by Large Language Models to guide smaller cross-encoder models. |
| Outcome: | The proposed model achieves ROC-AUC improvements of 1.4% on 9 multilingual e-commerce datasets, 2.4% on 3 ESCI datasets and 6% on GLUE datasets while being 50 times faster per sample. |
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) are prone to overfitting to training datasets . recent advances in NLP have shown that using PLMs such as BERT and RoBERTa on downstream tasks is effective. |
| Approach: | They propose a consistency-regularized knowledge distillation method which mitigates overfitting of existing methods. |
| Outcome: | The proposed method outperforms existing methods on the GLUE benchmark and synthetic datasets. |
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | a lack of standard evaluation metrics and benchmarks makes it difficult to identify strengths of Vietnamese NLP models. |
| Approach: | They propose to establish a standardized set of benchmarks for Vietnamese NLU . they propose to evaluate Vietnamese language understanding models using a pre-trained model . |
| Outcome: | The proposed model combines proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. |
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020.acl-demos)
Copied to clipboard
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman
| Challenge: | jiant is an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Approach: | They introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Outcome: | The proposed toolkit reproduces published performance on GLUE and SuperGLUE tasks. |
Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that transformer-based language models can achieve state-of-the-art performance on various natural language processing tasks. |
| Approach: | They propose a GPU-friendly 2:4 fine-grained structured sparsity and quantization scheme to maximize the GPU's acceleration constraints. |
| Outcome: | The proposed scheme achieves state-of-the-art compression on TLM model of various encoder and decoder blocks with negligible accuracy degradation on SQuAD, GLUE, CNN-DM & XSum and WikiText benchmarking tasks. |
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020.emnlp-demos)
Copied to clipboard
| Challenge: | TextAttack provides implementations of 16 adversarial attacks from the literature and supports a variety of models and datasets. |
| Approach: | They introduce a Python framework for adversarial attacks, data augmentation, and adversarially training in NLP. |
| Outcome: | This paper introduces a Python framework for adversarial attacks, data augmentation, and adversarially training in NLP. |
AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models (2024.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have demonstrated commendable performance on various NLP tasks. |
| Approach: | They propose a Parameter-Efficient Fine-Tuning (PEFT) method that incrementally freezes low-rank matrices during fine-tuning to reduce computation and alleviate over-fitting. |
| Outcome: | The proposed method achieves state-of-the-art performance with an average improvement of 0.85% on the GLUE benchmark while yielding up to 1.86 improvement as opposed to similar PEFT alternatives. |
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm (2022.acl-long)
Copied to clipboard
Shaoyi Huang, Dongkuan Xu, Ian Yen, Yijue Wang, Sung-En Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Sanguthevar Rajasekaran, Hang Liu, Caiwen Ding
| Challenge: | Conventional wisdom in pruning Transformer-based language models is that it reduces model expressiveness, but new research shows pruning increases risk of overfitting when performed at the fine-tuning phase. |
| Approach: | They propose to reduce pruning risk under pretrain-and-finetune paradigm . they propose to use knowledge distillation to improve pruning performance . |
| Outcome: | The proposed method outperforms the leading competitors on the GLUE benchmark. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches for positional dependencies do not satisfy all criteria for optimal position encoding. |
| Approach: | They propose a translation-invariant self-attention approach that accounts for relative position between tokens in an interpretable fashion without conventional embeddings. |
| Outcome: | The proposed model improves on regular ALBERT on GLUE tasks while adding orders of magnitude less positional parameters. |
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing studies on the performance of pre-trained language models on natural language understanding tasks have focused on the natural language inference and textual entailment tasks. |
| Approach: | They propose to use corrupted data to fine-tune pre-trained language models to assess their language understanding capabilities. |
| Outcome: | The proposed transformations can be applied to all but one NLU task and show that understanding the meaning of utterances is not required for high performance. |
Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models (2023.eacl-main)
Copied to clipboard
| Challenge: | In-distribution (ID) miscalibration and out-of-difference (OOD) detection are main concerns for pre-trained language models. |
| Approach: | They propose a triple-hybrid EBM which combines the benefits of classifier, conditional generative model and marginal generative models altogether. |
| Outcome: | The proposed model outperforms previous methods in terms of ID calibration and OOD detection by a large margin while maintaining competitive accuracy. |
Fine-Tuning Pre-Trained Language Models with Gaze Supervision (2024.acl-short)
Copied to clipboard
| Challenge: | Existing pre-trained language models lack a gaze module to exploit cognitive signals. |
| Approach: | They propose to integrate a gaze module into pre-trained language models at the fine-tuning stage to exploit cognitive signals. |
| Outcome: | The proposed model improves performance on the GLUE benchmark and standard fine-tuning and text augmentation baselines. |
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are the state-of-the-art (SOTA) models for natural language processing (NLP). |
| Approach: | They propose a patient and confident early exiting BERT (PCEE-BERT) that can work with different PLMs and popular model compression methods. |
| Outcome: | The proposed method outperforms existing models on the GLUE benchmarks and achieves different speed-up ratios. |
How effective is BERT without word ordering? Implications for language understanding and data privacy (2021.acl-short)
Copied to clipboard
| Challenge: | Ordered word sequences contain the rich structures that define language. |
| Approach: | They show that token representations and self-attention activations within BERT are resilient to shuffling the order of input tokens. |
| Outcome: | The proposed model is able to handle shuffled token representations and self-attention activations . the model can handle GLUE language understanding tasks with bag-of-words data . |
NeuronBlocks: Building Your NLP DNN Models Like Playing Lego (D19-3)
Copied to clipboard
| Challenge: | Deep Neural Networks (DNN) have been widely employed in industry to address various natural language processing tasks. |
| Approach: | They propose an NLP toolkit that encapsulates neural network modules as building blocks to construct various DNN models with complex architecture. |
| Outcome: | The proposed toolkit can build, train, and test various DNN models with complex architecture. |
When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Transfer learning (TL) in natural language processing has seen a surge of interest in recent years . pre-trained models have shown impressive ability to transfer to novel tasks . |
| Approach: | They compare two different methods of transfer learning in natural language processing to find out which is better. |
| Outcome: | The proposed methods perform better when the target task has fewer instances than the supporting task and vice versa. |
Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure (2026.eacl-srw)
Copied to clipboard
| Challenge: | a sparsity-exploiting backward pass is a memory-efficient way to accelerate LLM fine-tuning. |
| Approach: | They propose a method that exploits padding-induced gradient sparsity to accelerate backward computation. |
| Outcome: | The proposed method achieves a backward pass speedup of 2.15x on GLUE and 1.99x on reasoning benchmarks while maintaining memory usage identical to the regular PyTorch fine-tuning. |
AdaKron: An Adapter-based Parameter Efficient Model Tuning with Kronecker Product (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Pretrained Language Models (PLMs) have billions of parameters, causing computational challenges to fine-tuning models. |
| Approach: | They propose an Adapter-based fine-tuning with the Kronecker product that combine the outputs of two small networks to form a final vector whose dimension is the product of the dimensions of the individual outputs. |
| Outcome: | The proposed method achieves the same performance levels as state-of-the-art methods on the GLUE benchmark . |
Contrastive Distillation on Intermediate Representations for Language Model Compression (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to compress language models use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. |
| Approach: | They propose a method that uses knowledge distillation to distill knowledge through intermediate layers of the teacher via a contrastive objective. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark. |
Exploiting Language Model Prompts Using Similarity Measures: A Case Study on the Word-in-Context Task (2022.acl-short)
Copied to clipboard
| Challenge: | Existing few-shot approaches fail on the semantic distinction task of the Word-in-Context dataset. |
| Approach: | They propose a prompt-based approach which boosts few-shot performance to the level of fully supervised methods by using similarity metrics. |
| Outcome: | The proposed technique boosts few-shot performance to the level of fully supervised methods. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based pre-training models like BERT are computationally expensive and limited to resource-constrained devices. |
| Approach: | They propose a method which ternarizes the weights in a fine-tuned BERT model. |
| Outcome: | The proposed method outperforms the other methods on the GLUE and SQUAD benchmarks while being 14.9x smaller. |
AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have shown great power in natural language understanding . fine-grained tokenizations have advantages and disadvantages for learning of pre-tried models . |
| Approach: | They propose a pretrained language model based on both fine-grained and coarse-grain tokenizations . they propose to use both tokenization techniques to learn pre-trained models . |
| Outcome: | The proposed model outperforms BERT on benchmark datasets for Chinese and English . it can perform better with the same computational cost as BERT, the authors show . |
Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)
Copied to clipboard
| Challenge: | Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate. |
| Approach: | They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively. |
| Outcome: | The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines. |
EELBERT: Tiny Models through Dynamic Embeddings (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Empirical evaluation shows that the input embedding layer occupies a large portion of the model size. |
| Approach: | They propose an approach for compression of transformer-based models with minimal impact on downstream tasks by replacing the input embedding layer with dynamic embeddable computations. |
| Outcome: | Empirical evaluation shows that the proposed model is 15x smaller (1.2 MB) compared to the traditional model. |
Does Self-Attention Need Separate Weights in Transformers? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based models, respectively. |
| Approach: | They propose a simplified approach where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. |
| Outcome: | The proposed approach outperforms the BERT baseline on GLUE tasks even outperforming the standard BERT model in handling noisy and out-of-domain data. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Dynamic Data Selection for Curriculum Learning via Ability Estimation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing curriculum learning methods rely on heuristics to estimate difficulty of data . a major drawback is that they ignore competency of the model during training . |
| Approach: | They propose replacing difficulty heuristics with learned difficulty parameters . they propose a strategy that probes model ability at each training epoch . |
| Outcome: | The proposed strategy outperforms heuristic-based learning models on the GLUE classification tasks. |
Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling (2023.acl-long)
Copied to clipboard
| Challenge: | ensembling BERT models often improves accuracy but at the cost of significantly more computation and memory footprint. |
| Approach: | They propose a new ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. |
| Outcome: | The proposed method outperforms existing BERT models on GLUE and SuperGLUE with 100 training samples. |
BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the rise of many pre-trained language models (PLMs) such as GPT (Radford et al., 2019) and XLNet (Yang e.t al, 2019). |
| Approach: | They propose a framework which consists of two off-the-shelf methods for improving PLMs’ early exiting. |
| Outcome: | The proposed method can reduce the average latency of pre-trained language models and work with other inference speed-up methods like model pruning. |
Generate, Annotate, and Learn: NLP with Synthetic Text (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to generate unlabeled text are difficult to find. |
| Approach: | They propose a general framework called "generate, annotate, and learn" to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. |
| Outcome: | The proposed framework achieves state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard. |
Syntactic Structure Distillation Pretraining for Bidirectional Encoders (2020.tacl-1)
Copied to clipboard
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, Phil Blunsom
| Challenge: | Textual representation learners trained on large amounts of data have been successful on downstream tasks. |
| Approach: | They propose a knowledge distillation strategy for injecting syntactic biases into BERT pretraining by distilling the approximate marginal distribution over words in context from the syntaktic LM. |
| Outcome: | The proposed method reduces relative error by 2–21% on a diverse set of structured prediction tasks. |
TADA : Task Agnostic Dialect Adapters for English (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work on dialectal English NLP is task-specific, using manual annotated dialect data, weak supervision, or data augmentation. |
| Approach: | They propose a method for task-agnostic dialect adaptation by aligning non-SAE dialects with task-specific adapters from SAE. |
| Outcome: | The proposed method improves dialectal robustness on 4 dialectal variants of the GLUE benchmark without task-specific supervision. |
XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding (2022.acl-short)
Copied to clipboard
| Challenge: | XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks. |
| Approach: | They propose to distill visual information from pretrained multimodal transformers to pretrained language encoders to cater to the language-heavy characteristics of NLU. |
| Outcome: | The proposed framework outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG), and readability benchmarks. |
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Spectral-normalized identity priors (SNIP) is a structured pruning approach for a Transformer model. |
| Approach: | They propose a structured pruning approach which penalizes an entire residual module toward an identity mapping. |
| Outcome: | The proposed method improves on 5 GLUE benchmark tasks while maintaining comparable performance. |
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one. |
| Approach: | They propose a framework to assess adversarial robustness of multiple KD algorithms. |
| Outcome: | The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods. |
Efficient Learning of Multiple NLP Tasks via Collective Weight Factorization on BERT (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune a model for multiple tasks require a large amount of memory and computing power. |
| Approach: | They propose to factorize the weighs of a pre-trained Transformer model to improve training efficiency across multiple tasks by using BERT-Large as an instantiation of the Transformer and the GLUE as the evaluation benchmark. |
| Outcome: | The proposed method matches or improves the original fine-tuned model’s performance for each task while effectively decreasing parameter requirements by two orders of magnitude. |
Analysis and Prediction of NLP Models via Task Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Pretrained transformer-based encoders can be used to acquire rich text representations but need additional task supervision to be useful for downstream tasks. |
| Approach: | They propose a transformer to each MetaEval task and a neural network with a weighted encoder to perform the embeddings. |
| Outcome: | The proposed model outperforms baselines on GLUE tasks and can be used as a benchmark for future transfer learning research. |
Multi-Granularity Structural Knowledge Distillation for Language Model Compression (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to transfer knowledge to a small model are not enough to represent the rich semantics of a text. |
| Approach: | They propose to distill the knowledge to a student hierarchically across layers using a large teacher-student framework. |
| Outcome: | Experimental results show that the proposed method outperforms distillation methods on GLUE benchmark. |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
On Attention Redundancy: A Comprehensive Study (2021.naacl-main)
Copied to clipboard
| Challenge: | Attention redundancy has been observed among attention heads but has not been deeply studied in the literature. |
| Approach: | They propose a multi-layer multi-head self-attention mechanism which is widely applied in modern neural language models. |
| Outcome: | The proposed model is useful for interpretation and model compression. |
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for finetuning pretrained language models (PLMs) have risks in overfitting the pretraining tasks and data, which may lead to suboptimal performance. |
| Approach: | They propose a method which adds noise to parameters of PLMs before fine-tuning. |
| Outcome: | The proposed method can be used on GLUE English and XTREME multilingual benchmarks. |
RealFormer: Transformer Likes Residual Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing techniques to create Residual Attention Layer Transformer networks outperform the canonical Transformer on a wide spectrum of tasks. |
| Approach: | They propose a technique to create Residual Attention Layer Transformer networks that outperform the canonical Transformer on a wide spectrum of tasks. |
| Outcome: | The proposed technique outperforms the canonical Transformer on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQUAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. |
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing DA methods naively add a certain number of augmented samples without considering the quality and the added computational cost of these samples. |
| Approach: | They propose a data-augmented DA technique that generates or reweights augmented samples . they say it is faster to train and can be plugged into any DA method . |
| Outcome: | The proposed technique is faster to train and more efficient than existing methods. |
Compressing Pre-trained Language Models by Matrix Decomposition (2020.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models reach state-of-the-art results when fine-tuned individually; They also come with a significant memory and computational requirements, calling for methods to reduce model sizes (green AI). |
| Approach: | They propose a two-stage model-compression method to reduce a model’s inference time cost by decompressing the model into smaller matrices and performing feature distillation on the internal representation. |
| Outcome: | The proposed method reduces the number of parameters by 0.4x and increases inference speed by 1.45x while preserving the information contained within the model. |
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that Transformer Encoder model can't automatically capture word order, so explicit position embeddings are required to be fed into the target model. |
| Approach: | They propose a Transformer-based language model DecBERT that uses a causal attention mask to capture word order. |
| Outcome: | The proposed model improves on the GLUE language understanding benchmark and accelerates the pre-training process. |
A Flexible Multi-Task Model for BERT Serving (2022.acl-short)
Copied to clipboard
| Challenge: | a proposed BERT-based multi-task framework is suitable for iterative and incremental development of the tasks. |
| Approach: | They propose an efficient BERT-based multi-task framework that is suitable for iterative and incremental development of the tasks. |
| Outcome: | The proposed framework achieves 99.6% of performance of the full fine-tuning method while reducing up to two thirds of its overhead. |
Minimal Distillation Schedule for Extreme Language Model Compression (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for teacher assistant-based distillation require multiple trials to find the optimal teacher assistant. |
| Approach: | They propose a method that allows scheduling of an optimal teacher assistant in just one trial . they show that student performance is positively correlated with the scale-performance tradeoff . |
| Outcome: | The proposed method can select the optimal teacher assistant in just one trial . it can be used to compare performance of student and teacher assistants on GLUE benchmarks. |
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2021.findings-acl)
Copied to clipboard
| Challenge: | In July 2019, RoBERTa was the first to surpass a human baseline on GLUE . since then, 13 more methods have outperformed humans on the GLu leaderboard . |
| Approach: | They found that 75% to 90% of correct predictions of BERT-based classifiers remain constant after input words are randomly shuffled. |
| Outcome: | The proposed model outperforms humans on GLUE and SQuAD 2.0. |
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work has explored the suitability of pre-trained language models in low resource settings with less than 1,000 training data points. |
| Approach: | They propose to use pool-based active learning to speed up training while keeping the cost of labeling new data constant. |
| Outcome: | The proposed model can be fine-tuned to optimize for low-resource settings while keeping the cost of labeling constant. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Label Anchored Contrastive Learning for Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | a novel approach to contrastive learning for language understanding is not fully explored . contrastive training has been widely applied to self-supervised representation learning . |
| Approach: | They propose a label anchored contrastive learning approach for language understanding using a class label. |
| Outcome: | The proposed approach improves on GLUE and CLUE benchmarks by 4.1% compared to the state-of-the-art approaches . the proposed approach also improves under the few-shot and data imbalance settings . |
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart
| Challenge: | Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping. |
| Approach: | They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models. |
| Outcome: | The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets. |
Metacognitive Prompting Improves Understanding in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in prompting have enhanced reasoning in logic-intensive tasks for LLMs, yet the nuanced understanding abilities of these models remain underexplored. |
| Approach: | They propose a strategy inspired by human introspective reasoning processes to enhance LLMs' understanding abilities. |
| Outcome: | The proposed method outperforms chain-of-thought prompting and its advanced versions on ten natural language understanding (NLU) datasets. |
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained MLMs can be used to generalize well to a wide range of tasks. |
| Approach: | They propose to append [MASK]s at a later layer to reduce sequence length for earlier layers. |
| Outcome: | The proposed method outperforms RoBERTa for 6 out of 8 GLUE tasks on average by 0.4%. |
Structured Pruning Learns Compact and Accurate Models (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have high costs in terms of storage, memory, and computation time. |
| Approach: | They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency. |
| Outcome: | The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data. |
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training with such data. |
| Approach: | They propose a method to improve gender fairness of pre-trained models with less forgetting by evaluating them with general NLP tasks in GLUE. |
| Outcome: | The proposed method improves gender fairness of pre-trained models with less forgetting and performs better on GLUE by a large margin. |
Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent success of natural language processing (NLP) is driven by the adoption of large-scale pretrained language models. |
| Approach: | They propose a method to determine the impact of distillation influence on student generalization ability by prioritizing samples likely to enhance the student's generalization abilities. |
| Outcome: | The proposed method outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark. |
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn general representations of text can achieve sub-optimal performance in low-resource scenarios. |
| Approach: | They propose to use language model pre-training and multi-task learning to learn robust representations but these methods can achieve sub-optimal performance in low-resource scenarios. |
| Outcome: | The proposed model outperforms strong baselines on the GLUE benchmark and can be adapted to new tasks efficiently and effectively. |
COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing statically compressed pre-trained language models lack spatial and temporal efficiency due to their large size and wide width. |
| Approach: | They propose a spatially and temporally efficient model which retains the major capacity of PLMs. |
| Outcome: | The proposed model retains the major capacity of pre-trained language models at high compression and acceleration rate with 1/8 parameters and 1/19 FLOPs of BERT. |
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
Pruning Adatperfusion with Lottery Ticket Hypothesis (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive to fine-tune and require large storage. |
| Approach: | They propose a method to identify the influence of each adapter module and a way to prune adapters based on the Lottery Ticket Hypothesis. |
| Outcome: | The proposed model reduces size significantly while keeping performance intact. |
TextHide: Tackling Data Privacy in Language Understanding Tasks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Unsolved privacy challenges in distributed or federated learning are a challenge for many domains including Natural Language Processing. |
| Approach: | They propose a federated learning framework that adds an encryption step to prevent an eavesdropping attacker from recovering private text data. |
| Outcome: | The proposed model can effectively defend against attacks on shared gradients or representations and the averaged accuracy reduction is only 1.9%. |
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have shown remarkable results on various NLP tasks. |
| Approach: | They propose to improve the feed-forward network (FFN) in BERT with a higher computational cost than improving the multi-head attention (MHA). |
| Outcome: | The proposed model is 6.9 smaller and 4.4 faster than BERTBASE and has competitive performances on GLUE and SQuAD Benchmarks. |
Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations. |
| Approach: | They propose to build a sentence-level autoencoder from a pretrained transformer language model. |
| Outcome: | The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models. |
Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies describe how to apply contrastive learning to the language domain but it is difficult to apply data augmentation methods directly to language modeling. |
| Approach: | They propose a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. |
| Outcome: | The proposed method outperforms baseline models on sentence-level tasks with only 70% of memory compared to the baseline model. |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation (2023.findings-eacl)
Copied to clipboard
Mohammadreza Tayaranian Hosseini, Alireza Ghaffari, Marzieh S. Tahaei, Mehdi Rezagholizadeh, Masoud Asgharian, Vahid Partovi Nia
| Challenge: | Previously, researchers focused on lower bit-width integer data types for forward propagation and backward propagation of language models to save memory and computation. |
| Approach: | They use integer arithmetic for both forward and back propagation in the fine-tuning of BERT. |
| Outcome: | The proposed method improves on the GLUE and SQUAD benchmarks. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
A Study on How Attention Scores in the BERT Model Are Aware of Lexical Categories in Syntactic and Semantic Tasks on the GLUE Benchmark (2024.lrec-main)
Copied to clipboard
| Challenge: | In the realm of sentence comprehension, human attention is not evenly distributed across all words, indicating systematic variations in language processing. |
| Approach: | They propose to categorize tokens according to their lexical categories and focus on changes in attention scores among these categories during the fine-tuning process for downstream tasks. |
| Outcome: | The proposed model is based on a GLUE benchmark dataset and demonstrates that it assigns more bias to specific lexical categories irrespective of the task. |
VISPool: Enhancing Transformer Encoders with Vector Visibility Graph Neural Networks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing graph-based graph construction methods rely on static graphs and are not scalable with increasing document and word counts. |
| Approach: | They propose a dynamic graph construction method based on vector visibility graphs (VVGs) they propose scalable model architecture that integrates VVG convolutional networks into transformer pipelines. |
| Outcome: | The proposed model outperforms baseline models on the GLUE benchmark datasets. |
Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)
Copied to clipboard
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang
| Challenge: | Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized. |
| Approach: | They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions. |
| Outcome: | The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks. |
Gradient-based Intra-attention Pruning on Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and slow in inference due to their large sizes. |
| Approach: | They propose a structured pruning method which combines pruning with knowledge distillation to yield highly effective models. |
| Outcome: | The proposed method outperforms other pruning methods in sparsity regimes while maintaining 93% 99% performance. |
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent evidence shows that large-size pre-trained language models do not satisfy the logical negation property (LNP) However, their reliability is being challenged due to faulty behaviours and incomprehension on number-related representations. |
| Approach: | They propose a new intermediate training task to directly learn meaning text correspondence instead of relying on the distributional hypothesis. |
| Outcome: | The proposed approach outperforms previous models on 7 GLUE tasks and outperformed previous models. |
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing language pretraining frameworks only take the language context as selfsupervision . current frameworks do not take grounding information from the external visual world . |
| Approach: | They propose a visually-supervised language model that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to related images. |
| Outcome: | The proposed model improves on multiple pure-language tasks. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for creating vision-and-language models involve structural modifications and V&L pre-training. |
| Approach: | They propose to extend a language model through structural modifications and V&L pre-training to make it inherit the capability of natural language understanding from the original language model. |
| Outcome: | The proposed method improves performance of vision-and-language models by extending pre-trained models with the same pre-training. |
Span Fine-tuning for Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to fine-tune pre-trained language models are time-consuming and lack flexibility. |
| Approach: | They propose a span fine-tuning method which allows for a more efficient and efficient way of incorporating span-level information into pre-training. |
| Outcome: | Experiments on GLUE benchmark show that the proposed method significantly enhances the PrLM and offers more flexibility in an efficient way. |
UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)
Copied to clipboard
Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu
| Challenge: | Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data. |
| Approach: | They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts. |
| Outcome: | The proposed model breaks through performance upper bounds of experts without additional annotated data. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
Training Text-to-Text Transformers with Privacy Guarantees (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in NLP often stem from large transformer-based pre-trained models. |
| Approach: | They propose differentially private (DP) training as a potential mitigation for models that can memorize parts of training data. |
| Outcome: | The proposed model can memorize parts of training data and mitigate memorization concerns. |
BasqueGLUE: A Natural Language Understanding Benchmark for Basque (2022.lrec-1)
Copied to clipboard
| Challenge: | Natural Language Understanding (NLU) benchmarks are costly to develop and language-dependent . basqueGLUE is the first benchmark for Basque, a less-resourced language . |
| Approach: | They propose a benchmark for Basque, a less-resourced language, using existing datasets. |
| Outcome: | The proposed benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding beyond the detection of superficial clues. |
Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models address information confusion in position encoding and model representations. |
| Approach: | They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers. |
| Outcome: | The proposed techniques can improve pre-trained language models on GLUE benchmarks. |
An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | In the recent years, pre-trained language models have achieved great success in the NLP community. |
| Approach: | They propose two general strategies and an experimental procedure to troubleshoot HPO’s failure cases. |
| Outcome: | The proposed methods outperform grid search on two state-of-the-art language models using the same time budget and overfitting. |
BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks adapt from initial pretrained model to each downstream task directly, but ignore sequential nature of downstream tasks and feedback effect on pretrained models. |
| Approach: | They propose a framework to enable bidirectional knowledge transfer between pretrained models and downstream tasks in rounds. |
| Outcome: | The proposed framework improves on 9 GLUE datasets and 6 SuperGLUEs. |
Transformers with Learnable Activation Functions (2023.findings-eacl)
Copied to clipboard
| Challenge: | Activation functions can reduce the topological complexity of input data and improve model performance. |
| Approach: | They propose to consider data as a topology with its own shape to simplify its complexity and make it linearly separable in the output space. |
| Outcome: | The RAF-based Transformer model outperforms its FAF-based counterpart on the GLUE benchmark by 5.71 points and 2.05 points on SQuAD with all available data. |
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques to fine-tune pre-trained language models on downstream tasks are inadequate. |
| Approach: | They propose a technique to perturb hidden Transformers representations by enhancing generalization of hidden representations from different layers. |
| Outcome: | The proposed technique outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. |
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method for large language models. |
| Approach: | They propose a drop-in extension that reparameterizes a rank-rtot update as a sum of K *static* low-rank experts. |
| Outcome: | Experiments on reasoning and knowledge-intensive benchmarks show consistent gains over matched-budget LoRA. |
Continual and Multi-Task Architecture Search (P19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that architecture search can improve performance on language modeling and image classification tasks with reasonable training speed. |
| Approach: | They propose a continual architecture search approach that continually evolves the model parameters during sequential training of several tasks without losing performance on previously learned tasks. |
| Outcome: | The proposed approach improves language modeling and image classification with reasonable training speed and a weight-sharing strategy. |
SP3: Enhancing Structured Pruning via PCA Projection (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a widely used technique for reducing the size of pre-trained language models, but current methods overlook the potential of compressing the hidden dimension d in PLMs. |
| Approach: | They propose a structured pruning approach that projectes features into a space defined by principal components before masking the hidden dimension d in pre-trained language models. |
| Outcome: | Experiments on benchmarks show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy. |
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pruning results on benchmark transformers, such as BERT, are not as remarkable as those of convolutional neural networks. |
| Approach: | They propose to apply a knowledge-aware pruning process to transformer-based pre-trained language models to reduce model size and model weight. |
| Outcome: | The proposed pruning method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy. |
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach (2025.coling-main)
Copied to clipboard
| Challenge: | Using language models (LMs) has increased in use, and the use of biases and stereotypes is creating social problems. |
| Approach: | They propose a method to mitigate LM biases by continual training on biased data . they use masked language modeling to construct a Bias Vector as the difference between biased LMs and pre-trained LM weights . |
| Outcome: | The proposed method improves on the GLUE and SEAT benchmarks. |
Syntax-guided Contrastive Learning for Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies rely on additional syntax-driven attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. |
| Approach: | They propose a syntax-guided contrastive learning method which does not change the transformer architecture and does not alter the transformer structure. |
| Outcome: | The proposed method achieves consistent improvements in a variety of tasks including grammatical error detection, entity tasks, structural probing and GLUE. |
Partially-Random Initialization: A Smoking Gun for Binarization Hypothesis of BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained BERT has been used for natural language processing tasks but its performance is limited by memory and computational complexity. |
| Approach: | They propose to use pre-trained BERT to achieve decent accuracy . they propose to combine binary BERT with a randomly-initialized encoder . |
| Outcome: | The proposed model achieves state-of-the-art on GLUE and SQuAD benchmarks. |
Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations. |
| Approach: | They propose a representation learning approach that uses embeddings as anchors to model contextual representations. |
| Outcome: | The proposed model achieves 5x speedup and 1.2 points average improvement over MLM. |
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020.acl-main)
Copied to clipboard
| Challenge: | Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE . BERT is one of the largest models ever in NLP, but suffers from heavy model size and high latency . |
| Approach: | They propose a tool to compress and accelerate the popular BERT model by task-agnostic application. |
| Outcome: | The proposed model is 4.3x smaller and 5.5x faster than BERT_BASE . it achieves competitive results on well-known benchmarks . |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) require vast datasets for pretraining, making it difficult to train LLMs from scratch for lowresource languages. |
| Approach: | They propose to transform a language of the GLUE benchmark and then fine tune a pretrained model on that dataset. |
| Outcome: | The proposed models recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. |
On Surgical Fine-tuning for Language Encoders (2023.findings-emnlp)
Copied to clipboard
Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan
| Challenge: | preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements. |
| Approach: | They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning. |
| Outcome: | The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders. |
WYWEB: A NLP Evaluation Benchmark For Classical Chinese (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for classical Chinese are inadequate to evaluate performance of different NLP models. |
| Approach: | They propose an evaluation benchmark for classical Chinese NLP, which evaluates existing models. |
| Outcome: | The proposed benchmark evaluates the performance of existing models in classical Chinese. |
Knowledge Rumination for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models lack the capacity to handle knowledge-intensive tasks alone. |
| Approach: | They propose a new paradigm to help pre-trained language models utilize latent knowledge without retrieving it from external corpus. |
| Outcome: | The proposed paradigm can be applied to pre-trained language models without retrieving external knowledge from the corpus. |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods combine quantization with parameter-efficient fine-tuning but fail to meet practical performance requirements. |
| Approach: | They propose a measure and moment approach to optimize objective function for superior fine-tuning results by scaling the update process through a gradient. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on tasks like text generation, summarization, and understanding. |
Annealing Knowledge Distillation (2021.eacl-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful model compression technique for deep neural networks. |
| Approach: | They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally. |
| Outcome: | The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark. |
Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations. |
| Approach: | They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training. |
| Outcome: | The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy. |
Training ELECTRA Augmented with Multi-word Selection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-training methods for NLP tasks require massive computation resources. |
| Approach: | They propose a method that trains a discriminator to detect replaced tokens and select original tokens from candidate sets. |
| Outcome: | The proposed method improves ELECTRA based on multi-task learning on GLUE and SQUAD datasets. |
FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters. |
| Approach: | They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information. |
| Outcome: | The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency. |
GEM: A General Evaluation Benchmark for Multimodal Tasks (2021.findings-acl)
Copied to clipboard
Lin Su, Nan Duan, Edward Cui, Lei Ji, Chenfei Wu, Huaishao Luo, Yongfei Liu, Ming Zhong, Taroon Bharti, Arun Sacheti
| Challenge: | Existing datasets that focus on natural language tasks are not considered as a general evaluation benchmark for multimodal tasks. |
| Approach: | They present a general evaluation benchmark for multimodal tasks, GEM 1 . they compare it with existing multimodal vision-language datasets . |
| Outcome: | The proposed model is compared with existing vision-language datasets focusing on natural language tasks . it is the largest vision-linguistic dataset covering image-language tasks and video-language task at the same time . |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
LeeBERT: Learned Early Exit for BERT with cross-level optimization (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are resource exhaustive and computationally expensive for industrial scenarios. |
| Approach: | They propose a learning scheme to learn from each other to speed up inference . they ask each exit to learn the weights of different loss terms, instead of learning only from the last layer . |
| Outcome: | The proposed scheme improves state-of-the-art (SOTA) early exit methods for pre-trained models on the GLUE benchmark. |
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance. |
| Approach: | They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters. |
| Outcome: | The proposed method surpasses existing methods on the GLUE benchmark by a significant margin. |
A Simple and Effective Positional Encoding for Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that relative position encodings provide better performance than absolute position coding. |
| Approach: | They propose a mechanism to encode position and segment information into Transformer models using relative position encodings. |
| Outcome: | The proposed method achieves faster training and inference time while achieving competitive performance on GLUE, XTREME and WMT benchmarks. |
PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | A wide range of NLP tasks benefit from fine-tuning of pretrained language models (PLMs), however, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine- tuned model. |
| Approach: | They propose a noisy training mechanism which considers each parameter’s importance in the downstream task to help fine-tune pretrained language models. |
| Outcome: | The proposed method can be used to fine-tune pretrained language models on a wide range of tasks and consistently achieve higher performance. |
GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the early exiting of natural language processing (NLP) are notoriously gigantic and slow in both training and inference. |
| Approach: | They propose a framework for improving the early exiting of BERT by asking each exit to distill knowledge from each other. |
| Outcome: | The proposed framework outperforms the state-of-the-art (SOTA) BERT early exiting methods on the GLUE benchmark. |
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have been proposed and applied to many NLP tasks, yielding state-of-the-art performance, but high storage and computational costs obstruct them to be effectively deployed on resource-constrained devices and real-time applications. |
| Approach: | They propose a BERT distillation method which allows each intermediate student layer to learn from any intermediate teacher layers. |
| Outcome: | The proposed method can learn from different teacher layers adaptively for different NLP tasks. |
Posterior Calibrated Training on Sentence Classification Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for posterior calibration have been used to correct poorly calibrated posterior probabilities. |
| Approach: | They propose a posterior calibration procedure that optimizes posterior probability distributions while minimizing calibration errors. |
| Outcome: | The proposed procedure reduces calibration error and improves performance on both objectives. |
Robust Encodings: A Framework for Combating Adversarial Typos (2020.acl-main)
Copied to clipboard
| Challenge: | Existing procedures to defend against adversarial perturbations of inputs are heuristic in nature and susceptible to stronger attacks. |
| Approach: | They propose a framework that provides robustness without compromises on model architecture. |
| Outcome: | The proposed framework achieves robustness on six tasks while incompatible with state-of-the-art models like BERT. |
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)
Copied to clipboard
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan
| Challenge: | Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner. |
| Approach: | They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding. |
| Outcome: | The proposed benchmark compares the performance of AI systems on eight different tasks. |
Task-guided Disentangled Tuning for Pretrained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models are fine-tuned on task-specific datasets, but fail to capture task- specific patterns. |
| Approach: | They propose a method which disentangles task-relevant signals from entangled representations. |
| Outcome: | The proposed method improves generalization of representations by disentangling task-relevant signals from the entangled representations. |
Lifting the Curse of Capacity Gap in Distilling Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained language models require a tremendous amount of inference compute to perform. |
| Approach: | They propose to compress pretrained language models to small ones with a teacher-student paradigm to fill the capacity gap. |
| Outcome: | The proposed model achieves state-of-the-art performance at small FLOPs compared with competitive baselines. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
schuBERT: Optimizing Elements of BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer based models have achieved state-of-the-art performance for many natural language processing tasks including machine translation, question-answering tasks and semantic role labeling. |
| Approach: | They propose to reduce the number of parameters of BERT to obtain a much efficient light model. |
| Outcome: | The proposed model achieves 6.6% higher average accuracy on GLUE and SQuAD datasets than the previous model with three encoder layers while having the same number of parameters. |
Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Acceptability judgments are the most significant source of data in linguistics . however, there are still many open issues regarding methods for collecting and evaluating them. |
| Approach: | They propose to create a corpus of sentences with acceptability judgments using the same approach and the same steps as the English corpus. |
| Outcome: | The proposed corpus contains almost 10,000 sentences with acceptability judgments. |
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. |
| Approach: | They compare accuracy vs. model size tradeoffs using quantization and distillation methods . they find that pruning provides greater benefit than quantization . |
| Outcome: | The proposed methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. |
VALUE: Understanding Dialect Disparity in NLU (2022.acl-long)
Copied to clipboard
| Challenge: | English Natural Language Understanding systems outperform humans on benchmarks like GLUE and SuperGLUE, but they only use textbook Standard American English (SAE) . fewer studies have considered the effects of dialectal differences on performance . |
| Approach: | They propose a benchmark to evaluate the performance of English natural language understanding systems using a set of lexical and morphosyntactic transformation rules. |
| Outcome: | The proposed model outperforms humans on GLUE and SuperGLUE, but only on standard American English . the proposed model recruits fluent speakers of African American vernacular english to validate each feature transformation . |
Sparse Teachers Can Be Dense with Knowledge (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from a teacher of large scale to a student of smaller scale are limiting in overall knowledgeableness. |
| Approach: | They propose a sparse teacher trick to remove over-parameterized teachers that produce student-unfriendly knowledge and thus limit overall knowledgeableness. |
| Outcome: | The proposed trick removes the parameters that result in student-unfriendliness and leads to compelling performance in comparison with baselines. |
Aligned Weight Regularizers for Pruning Pretrained Neural Networks (2022.findings-acl)
Copied to clipboard
| Challenge: | Pruning aims to reduce the number of parameters while maintaining performance close to the original network. |
| Approach: | They propose a self-distilled pruning strategy that maximizes representational similarity between pruned and unpruned networks. |
| Outcome: | The proposed pruning strategy outperforms smaller models and outperformed smaller ones with an equal number of parameters and is competitive against (6 times) larger distilled networks. |
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is used to compress the pre-training and task-specific fine-tuning phases of large neural language models. |
| Approach: | They propose a sample-wise loss weighting method that re-weights the two losses for each sample. |
| Outcome: | The proposed method outperforms existing methods on 7 datasets of the GLUE benchmark. |
LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency (2022.coling-1)
Copied to clipboard
| Challenge: | Existing parameter-efficient methods focus on reducing trainable parameters but neglect the inference speed, which limits the ability to deploy PLMs. |
| Approach: | They propose to use a hypernetwork-assisted inter-layer connector to enhance inference efficiency by tuning parameters inside a linear connector between two Transformer layers. |
| Outcome: | The proposed model reduces model parameters to 11.75% while preserving performance degradation to less than 5%. |
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. |
| Approach: | They propose to decouple the feed-forward networks of the Transformer architecture into two parts to maintain old-domain knowledge and a mixture-of-adapters gate to inject domain-specific knowledge in parallel. |
| Outcome: | The proposed method achieves superior performance on in-domain, out-of-domain and knowledge-intensive tasks. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained large-scale language models have been criticized for their limited weight storage and computational speed on hardware platforms. |
| Approach: | They propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. |
| Outcome: | The proposed model achieves 5.0x accuracy on GLUE benchmarks and 1.79x compression rate on DistilBERT. |
SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | SupCL-Seq extends contrastive learning from computer vision to sequence classification tasks. |
| Approach: | They propose a supervised alternative to Masked Language Modeling (MLM) that extends contrastive learning to sequence optimization in NLP by altering the dropout mask probability in standard Transformer architectures. |
| Outcome: | The proposed method leads to large gains on the GLUE benchmark, including 6% absolute improvement on CoLA, 5.4% on MRPC, 4.7% on RTE and 2.6% on STS-B. |
G-Tuning: Improving Generalization of Pre-trained Language Models with Generative Adversarial Network (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical evaluations on the GLUE benchmark demonstrate that fine-tuning can enhance the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
| Approach: | They propose a fine-tuning framework that transforms the latent representation of pre-trained language models from a universal space to a target space and integrates a generative adversarial network into the fine-untun process. |
| Outcome: | Empirical evaluations on the GLUE benchmark and two additional demanding scenarios show that the proposed framework can improve the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token Pruning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to accelerate inference speed are model compression and dynamic computation (e.g., dynamic token pruning). |
| Approach: | They propose a two-stage knowledge distillation framework that produces a customized small language model for dynamic token pruning. |
| Outcome: | The proposed framework can make the small language model more customized for dynamic token pruning and achieve better speed-performance trade-off. |
MaskLoRA: Low-Rank Subspace–Induced Token Masking for Efficient and Faithful Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | MASKLORA is a plug-and-play masking mechanism that can be used to mask lowrank subspaces. |
| Approach: | They propose a plug-and-play masking mechanism that transforms PEFT's lowrank subspace into a faithful token selector. |
| Outcome: | The proposed masking mechanism matches full-model accuracy while yielding 1.3-2.6 speedups. |
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pretrained language models suffer from poor generalization . however, they add a perturbation to each model parameter equally, which is sub-optimal . |
| Approach: | They propose a sharpness-aware minimization optimization procedure that introduces a Fisher mask to improve the efficiency of SAM. |
| Outcome: | The proposed method outperforms the vanilla sharpness-aware minimization method on GLUE and SuperGLUE benchmarks. |
TAG: Gradient Attack on Transformer-based Language Models (2021.findings-emnlp)
Copied to clipboard
Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, Caiwen Ding
| Challenge: | Recent studies show that publicly shared gradients in the training process can reveal the private training data to a third-party. |
| Approach: | They propose a gradient attack algorithm to reconstruct the local training data using GLUE benchmarks. |
| Outcome: | The proposed algorithm achieves 1.5x recover rate and 2.5x ROUGE-2 over previous methods without the need of ground truth label. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
JGLUE: Japanese General Language Understanding Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | There is no benchmark for Japanese to evaluate and analyze NLU ability from different perspectives. |
| Approach: | They build a Japanese NLU benchmark from scratch without translation to measure general NLU ability in Japanese. |
| Outcome: | a Japanese NLU benchmark is built from scratch without translation to measure general NLU ability in Japanese. |
Causal Distillation for Language Models (2022.naacl-main)
Copied to clipboard
Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman
| Challenge: | Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. |
| Approach: | They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO). |
| Outcome: | The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003. |
FNet: Mixing Tokens with Fourier Transforms (2022.naacl-main)
Copied to clipboard
| Challenge: | Using simple linear transformations, Transformer encoders can be sped up with limited accuracy costs by replacing the self-attention sublayers with simple linear mixing mechanisms. |
| Approach: | They propose to replace the self-attention sublayer with a linear transformation that "mixes" input tokens. |
| Outcome: | The proposed model outperforms the “efficient Transformers” on the GLUE benchmark at longer input lengths and on smaller models with a light memory footprint. |
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered. |
| Approach: | They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language. |
| Outcome: | The proposed model performance is correlated with pre-training perplexity and performance. |
Imagination-Augmented Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for Natural Language Understanding focus on textual signals, which hinders models from learning efficiently from limited data samples. |
| Approach: | They propose an Imagination-Augmented Cross-modal Encoder to solve natural language understanding tasks from a novel learning perspective. |
| Outcome: | The proposed learning paradigm bridges the gap between human and agent language understanding in both linguistic and perceptual procedures. |
Towards More Efficient Post-training via Fourier Domain Adapter Framework (2025.findings-emnlp)
Copied to clipboard
| Challenge: | FDA reparameterizes the core projection operation of the adapter module directly in the Fourier domain. |
| Approach: | They propose a framework that reparameterizes the core projection operation of the adapter module directly in the Fourier domain. |
| Outcome: | The proposed framework outperforms existing parameter-efficient fine-tuning methods on GLUE, E2E NLG, and instruction tuning benchmarks. |
Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Transformer-based pre-trained models achieve state-of-the-art results, but they can be prohibitively costly. |
| Approach: | They propose a fine- and coarse-granularity hybrid self-attention that shortens the computational sequence length in self- attention by progressively shortening the computational time. |
| Outcome: | The proposed model reduces computation cost by shortening the computational sequence length in self-attention. |
BinaryBERT: Pushing the Limit of BERT Quantization (2021.acl-long)
Copied to clipboard
| Challenge: | Recent pre-trained language models have achieved remarkable performance improvement in various tasks, but the improvement generally comes at the cost of increasing model size and computation. |
| Approach: | They propose a binary quantization technique which initializes binaryBERT by splitting from a ternary network. |
| Outcome: | The proposed model achieves state-of-the-art performance on the GLUE and SQUAD benchmarks while being 24x smaller. |
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021.acl-long)
Copied to clipboard
| Challenge: | Using data augmentation to fine-tune pre-trained models with task-specific data has been shown to be ineffective and redundant during fine-timing. |
| Approach: | They propose a data augmentation technique to regularize pre-trained models and encourage them to learn more generalizable features by dropping contiguous spans during training. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and consistently exhibits superior generalization performances on out-of distribution and challenging counterexamples. |
SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embedding models lack interpretability for words . |
| Approach: | They propose to add interpretability to word embeddings by using a POLAR framework that enables wordsense aware interpretations for pre-trained contextual word embeds. |
| Outcome: | The proposed framework achieves comparable performance to existing embeddings across GLUE and SQuAD benchmarks. |
SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics (2024.naacl-long)
Copied to clipboard
Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, Koushik Sen
| Challenge: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
| Approach: | They propose a tool that analyzes transformer-based models and freezes less-contributory layers during fine-tuning to reduce the overall on-device memory usage. |
| Outcome: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models can be fine-tuned on limited training data, which can overfit and thus diminish performance. |
| Approach: | They propose a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout. |
| Outcome: | The proposed method outperforms existing methods on the GLUE benchmark and exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios. |
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (N19-1)
Copied to clipboard
| Challenge: | Multi-task learning is an inductive transfer mechanism that leverages information from related tasks to improve the primary model's generalization performance. |
| Approach: | They propose a multitask learning pipeline that finds relevant auxiliary tasks and learns their mixing ratio. |
| Outcome: | The proposed model can find relevant auxiliary tasks and learn their mixing ratio . the proposed model achieves significant performance boosts on several primary tasks . |
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transferability estimation has been a topic of great interest in computer vision fields . a lack of a comprehensive comparison between these estimation methods is a problem . |
| Approach: | They conduct a thorough survey of existing methods to find the most suitable model . they also outline difficulties of consideration of training details and applicability to text generation . |
| Outcome: | The proposed methods perform well with superiorities in effectiveness and efficiency. |
E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | E-LANG is an efficient method for building large and highly capable language models . existing methods are only applicable to encoder-only backbones and classification tasks . |
| Approach: | They propose an efficient dynamic inference approach which distributes inference between large accurate Super-models and light-weight Swift models. |
| Outcome: | The proposed method outperforms existing methods on GLUE, SuperGLUE and WMT with 3.3X computation speed and 2.9X computation cost. |
Effective Attention Sheds Light On Interpretability (2021.findings-acl)
Copied to clipboard
| Challenge: | Using a subset of the GLUE tasks and BERT, we compare the two attention matrices and show that their interpretations differ. |
| Approach: | They propose to use visualizing effective attention to interpret a transformer's behavior since it is more pertinent to the model output by design. |
| Outcome: | The proposed method is more relevant to the model output by design than visualizing attention weights. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
KinyaBERT: a Morphology-aware Kinyarwanda Language Model (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT are sub-optimal at handling morphologically rich languages. |
| Approach: | They propose a two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphology in a low-resource Kinyarwanda language. |
| Outcome: | The proposed model outperforms baseline models on the low-resource morphologically rich Kinyarwanda language by 2% in F1 score and 4.3% in average score of GLUE benchmark. |
TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices. |
| Approach: | They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages. |
| Outcome: | The proposed method accelerates inference and reduces model size while maintaining accuracy. |
Learning Task Sampling Policy for Multitask Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to train multi-task models with auxiliary tasks are limited by the number of combinations and the importance of each auxiliary task is not known a priori. |
| Approach: | They propose a search method that automatically assigns importance weights to auxiliary tasks to improve the target task quality. |
| Outcome: | The proposed method outperforms uniform sampling and the corresponding single-task baseline on XNLI and GLUE. |
Parameter-Efficient Transfer Learning with Diff Pruning (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained networks are difficult to deploy for multiple tasks in storage-constrained settings. |
| Approach: | Diff pruning enables parameter-efficient transfer learning that scales well with new tasks. |
| Outcome: | Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task. |
EmByte: Decomposition and Compression Learning for Small yet Private NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | EMBYTE is a byte-level tokenization model that reduces embedding parameters by up to 94% . it is also resilient to privacy threats such as gradient inversion attacks . |
| Approach: | EMBYTE is a byte-level tokenization model that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. |
| Outcome: | EMBYTE achieves substantial embedding compression while preserving accuracy and enhancing privacy. |
WARP: Word-level Adversarial ReProgramming (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning from pretrained language models are frozen and a task-specific head is trained on top of them. |
| Approach: | They propose an alternative approach that trains one or more task-specific layers on top of the language model. |
| Outcome: | The proposed approach outperforms existing methods on the GLUE leaderboard with just 32 training samples. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
Flooding-X: Improving BERT’s Resistance to Adversarial Attacks via Loss-Restricted Fine-Tuning (2022.acl-long)
Copied to clipboard
Qin Liu, Rui Zheng, Bao Rong, Jingyi Liu, ZhiHua Liu, Zhanzhan Cheng, Liang Qiao, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing approaches to generating adversarial perturbations scale up the cost of training computational complexity by the number of gradient steps it takes to obtain the adversarials. |
| Approach: | They propose a flood method which aims at better generalization and a criterion to bring hyper-parameter-dependent flooding into effect with a narrowed-down search space by measuring how the gradient steps taken within one epoch affect the loss of each batch. |
| Outcome: | The proposed method improves BERT’s resistance to textual adversarial attacks by a large margin and achieves state-of-the-art robust accuracy on various text classification and GLUE tasks. |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
MuLD: The Multitask Long Document Benchmark (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing benchmarks for NLP focus on tasks for one or two sentences, but efficient techniques are needed for processing much longer sequences. |
| Approach: | They propose to modify existing NLP tasks to create a long document benchmark which requires models to successfully model long-term dependencies in the text. |
| Outcome: | The proposed benchmark is much more challenging than its ‘short document’ equivalents. |
Utility is in the Eye of the User: A Critique of NLP Leaderboards (2020.emnlp-main)
Copied to clipboard
| Challenge: | Performance-based evaluation has been at the expense of other attributes valued by the NLP community, such as compactness and energy efficiency. |
| Approach: | They propose to frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. |
| Outcome: | The proposed model size and energy efficiency benchmarks have been successful in driving the creation of more accurate models, but have been at the expense of other attributes valued by the NLP community. |
An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training large language models is a standard practice in the natural language processing community. |
| Approach: | They propose to use elastic weight consolidation to mitigate catastrophic forgetting when pre-trained large language models are evaluated on generic benchmarks. |
| Outcome: | The proposed model achieves state-of-the-art on out-of domain tasks with minimal pre-training . elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. |
Dynamic Transformers Provide a False Sense of Efficiency (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models typically lead to high computational cost during inference. |
| Approach: | They propose a slowdown attack framework that can reduce inference efficiency by 80% by leveraging existing adversarial attacks targeting model accuracy. |
| Outcome: | The proposed framework can reduce the efficiency of multi-exit models by 80% on average, validating its effectiveness and generalization ability. |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |
LIMIT-BERT : Linguistics Informed Multi-Task BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models are usually trained on large amounts of unlabeled text data. |
| Approach: | They propose a multi-task language representations learning framework for multi-linguistics tasks by Multi-Task Learning. |
| Outcome: | The proposed model outperforms the baseline Whole Word Masking BERT on both dependency and constituent syntactic/semantic parsing, GLUE benchmark, and SNLI task. |
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Prompt-based learning is an emerging paradigm for exploiting knowledge learned by a pretrained language model. |
| Approach: | They propose a method to automatically select label mappings for few-shot text classification with prompting. |
| Outcome: | The proposed method achieves competitive performance on the GLUE benchmark without human effort or external resources. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |
Token and Head Adaptive Transformers for Efficient Natural Language Processing (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT have shown significant accuracy improvements on various tasks, but their computational cost and memory footprint are prohibitive. |
| Approach: | They propose to extend Length Adaptive Transformer to extend the model to a token and head pruning scheme to optimize pruning efficiency. |
| Outcome: | The proposed model can compress and accelerate BERT-based models by fine-tuning and a token and head pruning scheme. |
SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task. |
| Approach: | They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines . |
| Outcome: | The proposed model improves sample-efficiency and performance over strong baselines. |
CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation (2022.coling-1)
Copied to clipboard
Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Phillippe Langlais, Pascal Poupart
| Challenge: | Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. |
| Approach: | They propose a data augmentation technique tailored for knowledge distillation based on contrastive loss to improve masked adversarial data augmented by intermediate layer matching. |
| Outcome: | The proposed technique outperforms state-of-the-art methods on the GLUE benchmark and in an out-of domain evaluation. |
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful tool for deep learning applications. |
| Approach: | They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher. |
| Outcome: | The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. |
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020.coling-main)
Copied to clipboard
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, Zhenzhong Lan
| Challenge: | Existing language evaluation benchmarks for English are limited to English . lack of such benchmarks makes it difficult to replicate success in other languages . |
| Approach: | They introduce a large-scale Chinese language understanding evaluation benchmark . the benchmark uses a set of current state-of-the-art pre-trained Chinese models . |
| Outcome: | The first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark is released . the benchmark evaluates models across a wide range of tasks on original Chinese text . existing language evaluation benchmarks are mostly limited to English . |
AESOP: Paraphrase Generation with Adaptive Syntactic Control (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for paraphrase generation use fixed syntactic structures for all input sentences. |
| Approach: | They propose to add syntactical control to a pretrained language model to generate fluent paraphrases using a retrieval-based selection module. |
| Outcome: | The proposed model achieves state-of-the-art on semantic preservation and syntactic conformation on two benchmark datasets with ground-truth syntaktic control from human-annotated exemplars. |
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings. |
| Approach: | They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student . |
| Outcome: | The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Targeted Adversarial Training for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing adversarial training approaches focus on making adversarials less expensive or regularizing rather than replacing the standard training objective. |
| Approach: | They propose an algorithm to introspect current mistakes and prioritize adversarial training steps to where the model errs the most. |
| Outcome: | The proposed algorithm improves adversarial training for natural language understanding by introspecting mistakes and prioritizing training steps to where the model errs the most. |
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models contain matching subnetworks that have similar transfer learning performance as the original PLM. |
| Approach: | They propose to prune matching subnetworks using magnitude-based pruning . they propose to optimize the subnetwork structure towards the pre-training objectives . |
| Outcome: | The proposed method is more efficient in searching subnetworks and advantageous when fine-tuning within a range of data scarcity. |
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient language model tuning (PELT) match the performance of fine-tuning with fewer trainable parameters. |
| Approach: | They propose a framework which integrates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. |
| Outcome: | The proposed framework outperforms fine-tuning methods on the GLUE benchmark and achieves 14% gains over the best individual PELT method. |
Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model Compression (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) can transfer knowledge from the original model into a compact model to achieve model compression. |
| Approach: | They propose a knowledge distillation method with reptile meta-learning to facilitate the transfer of knowledge from the teacher to the student. |
| Outcome: | Extensive experiments on the GLUE benchmark show the proposed method performs better than previous methods. |
How transfer learning impacts linguistic knowledge in deep NLP models? (2021.findings-acl)
Copied to clipboard
| Challenge: | Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. |
| Approach: | They propose to fine-tune pre-trained models towards downstream NLP tasks to capture linguistic knowledge. |
| Outcome: | The proposed model is adapted to GLUE tasks and retains linguistic information in the network while forgetting it. |
DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive. |
| Approach: | They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality. |
| Outcome: | The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters. |
Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training. |
| Approach: | They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model. |
| Outcome: | The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement) |
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks . |
| Approach: | They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost. |
| Outcome: | The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states . |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training . |
| Approach: | They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters. |
| Outcome: | The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT. |
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark (P19-1)
Copied to clipboard
| Challenge: | GLUE is a suite of language understanding tasks that has seen dramatic progress in the past year . average performance on the benchmark is 83.9, state of the art at the time of writing . |
| Approach: | They use crowdsourcing to measure human performance on a set of language understanding tasks and 20 examples to determine whether there is room for improvement. |
| Outcome: | The GLUE benchmark outperforms state-of-the-art models on six of the nine tasks and achieves an average score of 87.1. |
GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021.emnlp-main)
Copied to clipboard
| Challenge: | A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. |
| Approach: | They propose an automatic auxiliary task selection method based on gradient calculation in Transformer-based models that improves MT-DNN performance. |
| Outcome: | The proposed method improves MT-DNN performance on 8 natural language understanding (GLUE) tasks, while costing less than AUTOSEM and comparable GPU consumption. |
ParaMac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing unsupervised methods for paraphrase generation are weak in semantic equivalence or expression diversity. |
| Approach: | They propose a framework for unsupervised paraphrase generation that employs multi-aspect equivalence constraints and multi-granularity diversifying mechanisms to achieve good semantic equvalence and expressive diversity. |
| Outcome: | The proposed framework achieves 9.1% and 3.3% absolute gains over previous SOTA on Quora and MSCOCO and can improve to 18.0% and 4.6% on GLUE. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing paradigm to fine-tune parameters of pre-trained language models poses problems in data-scarce and resource-limited scenarios. |
| Approach: | They propose a parameter-efficient fine-tuning method HiFi that fine-tails only the highly informative and strongly correlated attention heads for the specific task. |
| Outcome: | The proposed method obtains state-of-the-art over the prior benchmarks on the GLUE benchmark. |
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources. |
| Approach: | They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism. |
| Outcome: | The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%. |
ltzGLUE: Luxembourgish General Language Understanding Evaluation (2026.findings-acl)
Copied to clipboard
Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cedric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
| Challenge: | ltzGLUE is the first official NLU benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. |
| Approach: | They propose a new natural language understanding (NLU) benchmark for Luxembourgish based on the popular GLUE benchmark for English. |
| Outcome: | The proposed model performs well across many languages and is based on the GLUE benchmark for English. |
Word Order Does Matter and Shuffled Language Models Know It (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. |
| Approach: | They propose a transformer-based BERT architecture that uses a fixed, sinusoidal position embedding added to each token embeddable to compensate for this absence of linear order. |
| Outcome: | The proposed model retains word order information because of the dependencies between sentence length and unigram probabilities. |
On Robustness of Finetuned Transformer-based NLP Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Transformer-based language models have been finetuned for a large number of tasks. |
| Approach: | They characterize changes between pretrained and finetuned models with CKA and STIR metrics. |
| Outcome: | The proposed models are more robust to perturbations than BERT and T5 on classification tasks and generation tasks. |
ParaTag: A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets only annotate a binary label for each sentence pair. Existing models only annnotate binary labels for each phrase pair. |
| Approach: | They propose a novel binary paraphrase classification task that annotates the degree of paraphrase between sentences and a new annotation schema that labels the minimum spans of tokens in a sentence that don't have the corresponding paraphrases in the other sentence. |
| Outcome: | The proposed dataset can be used to train an automatic scorer for language generation evaluation. |
R-AT: Regularized Adversarial Training for Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, adversarial training is a popular and powerful regularization method in the natural language domain. |
| Approach: | They propose to regularize adversarial training via dropout by perturbing word embeddings . they find that R-AT can improve many models by reducing adversariality . |
| Outcome: | The proposed method can reduce the inconsistency between training and testing of models with dropout. |
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)
Copied to clipboard
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
| Challenge: | XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios . |
| Approach: | They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora. |
| Outcome: | The proposed dataset is labeled in English and includes only natural language understanding tasks. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022.acl-long)
Copied to clipboard
| Challenge: | Existing work has resorted to sharing weights among models, but results are not affordable for real-world deployment. |
| Approach: | They propose a consistency-regularized ensemble learning approach based on perturbed models to retain ensemble benefits while maintaining a low memory cost. |
| Outcome: | The proposed approach outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches for language models have overlooked the difficulty of training examples. |
| Approach: | They propose a framework that controls difficulty of training examples during pre-training by a tutor network. |
| Outcome: | The proposed framework outperforms state-of-the-art KD methods with student models on the GLUE benchmark. |
Transkimmer: Transformer Learns to Layer-wise Skim (2022.acl-long)
Copied to clipboard
| Challenge: | Prior work has proposed to augment Transformer model with the capability of skimming tokens to improve its computational efficiency. |
| Approach: | They propose to add a parameterized predictor before each layer that learns to make the skimming decision. |
| Outcome: | The proposed model achieves 10.97x speedup on GLUE benchmark compared with BERT-base baseline with less than 1% accuracy degradation. |
SkipBERT: Efficient Inference with Shallow Layer Skipping (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have significant demands in computation and inference time, limiting their use in resource-constrained or latencysensitive applications. |
| Approach: | They propose to encode text chunks into independent representations and skip computation of shallow layers to accelerate inference. |
| Outcome: | The proposed approach can reduce latency by 65% without sacrificing performance. |
GhostBERT: Generate More Features with Cheap Operations for BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that some parameters in pre-trained language models can be pruned away without severe accuracy degradation. |
| Approach: | They propose a method which generates more features with very cheap operations from the remaining features and can be applied to unpruned BERT models to enhance their performance. |
| Outcome: | Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of the proposed method. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
Zhuocheng Gong, Jiahao Liu, Qifan Wang, Yang Yang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Rui Yan
| Challenge: | Quantization is a viable solution for pre-trained language models, but most existing methods are task-specific and require customized training and quantization with a large number of trainable parameters. |
| Approach: | They propose a "quantize before fine-tuning" framework that allows for quantization with a large number of trainable parameters on each individual task. |
| Outcome: | The proposed framework is compatible with quantization-aware training and post-training quantization and corrects quantization errors. |
IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for logical reasoning have surpassed the average performance of humans in many tasks like SQuAD but there is still a long way to go when it comes to logical reasoning. |
| Approach: | They propose an InDicator-Oriented Logic Pre-training task which logically strengthens pre-trained models with the help of 6 types of logical indicators and a logicalally rich dataset. |
| Outcome: | The proposed task achieves state-of-the-art on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC. |
Parameter-Efficient Tuning Makes a Good Classification Head (2022.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, pretrained models revolutionized the paradigm of natural language understanding . but the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning . |
| Approach: | They propose to append a randomly initialized classification head after the pretrained backbone and finetune the whole model. |
| Outcome: | The proposed classification head can be replaced with the randomly initialized heads for a stable performance gain. |
Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates (2021.acl-long)
Copied to clipboard
| Challenge: | Using negative flips, we quantify, reduce and analyze regression errors in deep neural networks. |
| Approach: | They propose to quantify, reduce and analyze regression errors in NLP models by negative flips. |
| Outcome: | The proposed model update regression has a prevalent presence across tasks in the GLUE benchmark. |
Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. |
| Approach: | They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
| Outcome: | The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
LPC: A Logits and Parameter Calibration Framework for Continual Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solve catastrophic forgetting problem are varied . current approaches to learn continuous learning are based on replay-based methods . |
| Approach: | They propose to calibrate parameters and logits so that preserving old parameters and generalized learning on new concepts can be solved simultaneously. |
| Outcome: | The proposed model achieves state-of-the-art performance in all scenarios. |
FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |
Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for lightweight fine-tuning are ineffective in low-resource settings but fail in high-resourced settings, leading to unreliable outcomes. |
| Approach: | They propose a calibration strategy that takes into account the inherent variance of generalization ability in model components and potential changes during the fine-tuning process. |
| Outcome: | The proposed calibration improves GLUE score by 3.1 points over the previous calibration method. |
HashFormers: Towards Vocabulary-independent Pre-trained Transformers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. |
| Approach: | They propose a family of vocabulary-independent pre-trained transformers that support unlimited vocabulary . they propose to map each token to its corresponding embedding by default . |
| Outcome: | The proposed models are more memory efficient than existing models while achieving comparable performance on multiple text classification tasks. |
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficient tuning (PET) methods fit pre-trained language models to downstream tasks by computing a small compressed update for a subset of parameter updates or appending and fine-tuning a few new parameters to the pre-train network. |
| Approach: | They propose a parameter-efficient tuning method that prunes PET architectures using structured pruning. |
| Outcome: | The proposed method performs well in practice but could be improved via NAS . it can be used to fine-tune large pre-trained language models on multiple tasks . |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
EsCoLA: Spanish Corpus of Linguistic Acceptability (2024.lrec-main)
Copied to clipboard
| Challenge: | Acceptability is one of the general language understanding evaluation benchmarks (GLUE) probing tasks . EsCoLA consists of 11,174 sentences and their acceptability judgements as found in well-known Spanish reference grammars. |
| Approach: | They propose to use a corpus of linguistic acceptability (ESCoLA) EsCoLA consists of 11,174 sentences and their acceptability judgements . |
| Outcome: | The proposed task is based on 11,174 sentences and their acceptability judgements as found in well-known Spanish reference grammars. |
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture semantic and syntactic information, but no study has examined how information loss in input token characters affects their performance. |
| Approach: | They address this gap by pre-training language models using small subsets of token characters. |
| Outcome: | The proposed model retains 90% and 77% of the full-token model in standard NLU benchmarks and probing tasks even under extreme settings. |
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory. |
| Approach: | They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix. |
| Outcome: | The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix. |
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Approach: | They propose a parameter-efficient fine-tuning approach that leverages a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Outcome: | The proposed approach outperforms existing methods on GLUE and E2E benchmarks and is effective in instruction-tuning large language models and image classification models. |
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. |
| Approach: | They propose a new compression paradigm that extracts knowledge from pre-trained language models to construct a knowledge store from which the model can leverage it for effective inference. |
| Outcome: | The proposed model extracts knowledge from LLMs to construct a knowledge store, which the model can leverage for effective inference. |
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing methods to train multi-task neural networks outperform or even match their single-task counterparts are difficult to implement. |
| Approach: | They propose a method that uses knowledge distillation to train multi-task neural networks that outperform or even match their single-task counterparts. |
| Outcome: | The proposed method outperforms or matches single-task neural networks on the GLUE benchmark. |
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models that use heuristics to shorten sequence lengths are computationally prohibitive. |
| Approach: | They propose a new method to shorten sequence lengths by transforming tokens through encoders and a core-set based token selection method that avoids expensive pre-training and fine tuning. |
| Outcome: | The proposed model outperforms existing models on GLUE benchmarks and Long Range Arena datasets and demonstrates that it is cost-effective and space-efficient. |
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate layer matching are limited due to huge over-parameterization . |
| Approach: | They propose to match intermediate layers of teacher and student in output space via attention-based layer projection. |
| Outcome: | The proposed method outperforms existing methods on GLUE tasks. |
Proxy Indicators for the Quality of Open-domain Dialogues (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of open-domain dialogues are expensive and require human annotators to evaluate their quality. |
| Approach: | They propose to use a deep-learning model trained on the general language understanding evaluation benchmark to serve as a quality indication of open-domain dialogues. |
| Outcome: | The proposed model can infer various quality metrics and derive a component-based overall score. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to compress neural networks by progressive module replacement is proposed . a number of techniques have been proposed to compress pretraining and fine-tuning models . |
| Approach: | They propose a model compression approach that divides BERT into modules and builds their compact substitutes. |
| Outcome: | The proposed approach outperforms existing knowledge distillation approaches on GLUE benchmark . it is based on a model that divides the original BERT into several modules and builds their substitutes . |
SlowBERT: Slow-down Attacks on Input-adaptive Multi-exit BERT (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models such as Google’s BERT have been gaining significant improvements to various down-stream applications, but the enormous training and inference cost severely hinders its practice on real-time applications and hardwareconstrained edge devices. |
| Approach: | They propose a slow-down attack on input-adaptive multi-exit BERT where the adversary imperceptibly modifies the input texts to drastically increase the inference cost. |
| Outcome: | The proposed attack on input-adaptive multi-exit BERT dramatically increases the average inference cost by 4.57, which would hurt the service quality of multi-extit BRT in practice, e.g., increasing the real-time cloud services’ response times for online users. |
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune deep pretrained language models face catastrophic forgetting problems. |
| Approach: | They propose a recall and learn mechanism which integrates pretraining and downstream tasks into a single mechanism. |
| Outcome: | The proposed method achieves state-of-the-art performance on the GLUE benchmark and better average performance than directly fine-tuning of BERT-large. |
Perturbation Augmentation for Fairer NLP (2022.emnlp-main)
Copied to clipboard
| Challenge: | Unwanted and often harmful social biases are becoming more salient in NLP research. |
| Approach: | They propose to train a neural perturbation model that rewrites demographic references in text to make them more fair. |
| Outcome: | The proposed model outperforms heuristic alternatives on a large dataset of human annotated text perturbations. |
Conceptor-Aided Debiasing of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models reflect inherent social biases of their training corpus. |
| Approach: | They propose two methods to identify and remove the bias subspace in pre-trained large language models such as BERT and GPT by applying conceptors to a conceptor NOT operation. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) debiasing while maintaining LLMs’ performance on the GLUE benchmark. |
How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is a widely used self-supervised pretraining objective. |
| Approach: | They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context. |
| Outcome: | The proposed objectives show that they should have half the complexity needed to perform comparably to MLM. |
LoRaDA: Low-Rank Direct Attention Adaptation for Efficient LLM Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in parameter-efficient fine-tuning techniques allow for adjustments to only a minor fraction of the parameters of language models. |
| Approach: | They propose a low-rank direct attention adapted method for efficient LLM fine-tuning . they propose LMAM, which can bring negative attention to self-attention modules . |
| Outcome: | The proposed method outperforms the full fine-tuning method by 2.1% on GLUE benchmark. |
MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process. |
| Approach: | They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance. |
| Outcome: | The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models. |
An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to explicitly incorporate human-defined linguistic priors into fine-tuning tasks. |
| Approach: | They replace parsed graphs or trees with trivial ones to investigate linguistic priors . they propose to use trivial graphs as baselines to design advanced knowledge fusion methods . |
| Outcome: | The use of trivial graphs improves performance in fully-supervised and few-shot settings. |
DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
| Approach: | They propose a framework that leverages singular value decomposition to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces. |
| Outcome: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks show that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
Robertha: Eigenspectrum Regularized Attention for Robust Natural Language Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing robustness methods sacrifice clean performance or fail to generalize to higher corruption levels. |
| Approach: | They propose a mechanism that uses semantic patterns to pull corrupted embeddings toward correct representations by Eigenspectrum Regularization. |
| Outcome: | The proposed mechanism outperforms robustness methods on 13 GLUE and SuperGLUE tasks while maintaining competitive clean performance. |
ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has shown significant practical benefits when using language models . current two stage MTL introduces a substantial number of additional parameters . |
| Approach: | They propose a multi-task learning method that leverages existing knowledge for a target task. |
| Outcome: | The proposed method outperforms baselines on three benchmarks and two encoder LMs with a small number of transfer parameters. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation use a two-stage paradigm: general distillation with a task-agnostic general corpus and task-specific distillation using augmented task- specific corpus. |
| Approach: | They propose a contextualized corpus that contextualizes task corpus with large-scale general corpus through relevance-based text retrieval to improve student learning. |
| Outcome: | The proposed model improves on the GLUE benchmark and shows that it is better than generalized corpus and augmented task-specific corpus. |
HuLU: Hungarian Language Understanding Benchmark Kit (2024.lrec-main)
Copied to clipboard
Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, Tamás Váradi
| Challenge: | The Hungarian Language Understanding (HuLU) benchmark is a comprehensive assessment framework designed to evaluate the performance of neural language models on Hungary language tasks. |
| Approach: | They propose to use a framework to evaluate the performance of neural language models on Hungarian language tasks. |
| Outcome: | The framework evaluates models against Hungarian language tasks using a web service and a leaderboard. |
With Little Power Comes Great Responsibility (2020.emnlp-main)
Copied to clipboard
| Challenge: | Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements and increase the chances of exaggerated findings. |
| Approach: | They characterize typical statistical power for a variety of settings and characterize it by a set of existing NLP papers and datasets. |
| Outcome: | The authors characterize typical power for a variety of settings and find it common in the literature. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |
Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pruning methods focus on a single pruning criterion and lack variety. |
| Approach: | They propose a model pruning strategy that generates several pruning masks randomly and then chooses the optimal mask from the pool of mask candidates. |
| Outcome: | The proposed pruning strategy achieves state-of-the-art performance across eight datasets from GLUE, particularly excelling at high levels of sparsity. |
Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases (2023.acl-long)
Copied to clipboard
| Challenge: | Existing debiasing techniques use Counterfactual Data Augmentation (CDA) to balance the training corpus, but this technique slightly modifies the original corpus limiting the representation distance between different demographic groups. |
| Approach: | They propose a two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation to mitigate social biases in PLMs’ encoding. |
| Outcome: | The proposed model outperforms baselines in terms of debiasing performance while maintaining the language modeling capability of PLMs. |
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
PAD-Net: An Efficient Framework for Dynamic Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Dynamic networks can significantly improve the model’s representation power with acceptable computational cost. |
| Approach: | They propose a partially dynamic network to transform redundant dynamic parameters into static ones and iterative mode partition to partition dynamic and static parameters efficiently. |
| Outcome: | The proposed network surpasses fully dynamic networks by +0.7% top-1 acc with only 30% dynamic parameters for DY-Conv and +1.9% average score in language understanding with only 50% dynamic parameters. |
Studying word order through iterative shuffling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models has made this hypothesis popular . but, word order is not important enough to make sentence structure relevant . |
| Approach: | They propose an efficient procedure that finds word order having highest likelihood under a fixed language model. |
| Outcome: | The proposed procedure can be used to find the ordering of a bag of words having the highest likelihood under a fixed language model. |
How to Train BERT with an Academic Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models such as BERT are used in many NLP tasks, but their pretraining phase can be prohibitively expensive for startups and academic research groups. |
| Approach: | They propose a recipe for pretraining a large language model in 24 hours using a low-end deep learning server. |
| Outcome: | The proposed model can be trained on GLUE tasks at fraction of the cost of pretraining. |
TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing PEFT methods can be costly and underfit token-level contexts. |
| Approach: | They propose a PEFT method that performs fine-grained, token-specific edits with a small additional inference overhead and minimal tuning. |
| Outcome: | The proposed method outperforms state-of-the-art methods in 8 tasks and GLUE with a minimal tuning overhead and inference overhead. |
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning (2023.acl-long)
Copied to clipboard
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
| Challenge: | XY-LENT: X-Y bitext enhanced Language ENcodings achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands. |
| Approach: | They propose a method for building multilingual representation models that are competitive with existing models and more parameter efficient. |
| Outcome: | The proposed model outperforms XLM-R XXL and is 5x and 6x smaller respectively. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
TLM: Token-Level Masking for Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Structured dropout approaches have been investigated to regularize the multi-head attention mechanism in Transformers. |
| Approach: | They propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting by manipulating the connections between tokens in the multi-head attention via masking. |
| Outcome: | The proposed regularization scheme outperforms attention dropout and DropHead on 18 datasets and can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). |
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments show that MEO significantly improves computational efficiency . compared to dense networks, sparsely activated networks only employ a few parameters for each input . |
| Approach: | They propose a method that merges multiple experts into one to reduce computation costs . they demonstrate that a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters for each input . |
| Outcome: | The proposed approach reduces the computational cost to that of a single expert by 83.3% compared to 82.6% in vanilla MoE. |
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have shown remarkable performance on diverse NLP tasks through pre-training and fine-tuning. |
| Approach: | They propose a numerically robust IM-connection incorporating a layer of BERT as a solution of Ordinary Differential Equations (ODEs) . Experimental results validate the robustness of IM BERT under various conditions. |
| Outcome: | The proposed model outperforms the existing model on the adversarial GLUE dataset by 5.9%p on low-resource scenarios. |
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scale pre-trained language models to a deeper model depth share all parameters or use extra blocks. |
| Approach: | They propose a parameter-efficient approach to scaling pre-trained language models to a deeper model depth using matrix product operator. |
| Outcome: | The proposed model scales pre-trained language models to a deeper model depth by 4x and achieves 0.1 points higher than BERT-large for GLUE score. |
A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are limited due to the need for increased computational resources. |
| Approach: | They propose a method to learn PEFT parameters from data by projecting high dimensional parameters onto low dimensional parameter manifolds or identifying PEFT parametrically. |
| Outcome: | The proposed method can be used to identify PEFT parameters on GLUE tasks. |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |
Representation Projection Invariance Mitigates Representation Collapse (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning contextualized representations by pre-trained models can lead to representation degradation, which can result in instability, sub-optimal performance, and weak generalization. |
| Approach: | They propose a regularization method to maintain the information content of representations and reduce representation collapse during fine-tuning. |
| Outcome: | The proposed method outperforms baselines on most tasks and improves out-of-distribution performance. |
Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) is a low-cost alternative to full fine-timing due to the massive overhead. |
| Approach: | They propose a Mixture-of-Experts approach that enhances specialization while maintaining low resource overhead. |
| Outcome: | The proposed approach outperforms or matches state-of-the-art methods on GLUE, GSM8K, MBPP, and a text rewriting task from SmolTalk. |
Stochastic Fine-Tuning of Language Models Using Masked Gradients (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the dominant paradigm in Natural Language Processing but fine-tuning them for specific downstream tasks often requires updating a vast number of parameters. |
| Approach: | They propose a method that selectively updates a small subset of parameters in each step of the tuning process. |
| Outcome: | The proposed approach outperforms existing fine-tuning methods while updating merely **0.08**% of the model’s parameters. |
Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. |
| Approach: | They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods. |
| Outcome: | The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance. |
NLPre: A Revised Approach towards Language-centric Benchmarking of Natural Language Preprocessing Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | GLUE benchmarking system enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Approach: | They propose a language-centric benchmarking system that enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Outcome: | The proposed system is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. |
GMFL: Efficient Global Masking for Federated LLM Fine-tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent solution to mitigate the communication and computation costs in federated fine-tuning of Large Language Models (LLMs). |
| Approach: | They propose a plug-and-play layer freezing mechanism to integrate with existing federated fine-tuning frameworks. |
| Outcome: | The proposed solution reduces communication overhead and lowers computational costs while preserving the performance of the underlying federated fine-tuning methods. |
Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Structured pruning is an effective technique for compressing pre-trained language models (PLMs), but it requires retraining, leading to additional computational overhead. |
| Approach: | They propose a task-specific pruning framework that prunes redundant modules of pre-trained language models before fine-tuning them. |
| Outcome: | The proposed pruning framework achieves higher performance on GLUE, SQUAD, WikiText-2, Wik-103, and PTB datasets while reducing the time required for fine-tuning. |
SLlama: Parameter-Efficient Language Model Architecture for Enhanced Linguistic Competence Under Strict Data Constraints (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) have shown remarkable performance across a wide array of tasks. |
| Approach: | They propose an architecture that preserves parameter efficiency of tied models without sacrificing representational benefits of untied embeddings. |
| Outcome: | The proposed architecture achieves a 31.72% improvement in linguistic knowledge acquisition over the baseline model. |
Sinkhorn Distance Minimization for Knowledge Distillation (2024.lrec-main)
Copied to clipboard
Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, Houqiang Li
| Challenge: | Existing knowledge distillation methods investigate divergence measures but fail to deliver effective supervision when few distribution overlap exists between teacher and student. |
| Approach: | They propose a knowledge distillation method that exploits the Sinkhorn distance to ensure a nuanced assessment of the disparity between teacher and student distributions. |
| Outcome: | The proposed method outperforms state-of-the-art methods on all kinds of LLMs with encoder-only, encoder decoder, and decoded architectures. |
GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. |
| Approach: | They propose a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. |
| Outcome: | Experiments show that GeLoRA outperforms adaptive LoRA methods by up to +1.0% . |
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-fine-tuned large language models (LLMs) under 14B parameters underperform on NLU tasks . we explore a framework to improve the NLU capabilities of LLMs . |
| Approach: | They propose to use Proximal Policy Optimization to improve NLU capabilities . they frame NLU as a reinforcement learning environment and optimize for reward signals . |
| Outcome: | The proposed framework outperforms supervised fine-tuning on GLUE and superGLUE tasks. |
Vygotsky Distance: Measure for Benchmark Task Similarity (2024.lrec-main)
Copied to clipboard
| Challenge: | GLUE, SuperGLUE and RussianSuperGLUE benchmarks are arbitrary sets of tasks that are not generalized. |
| Approach: | They propose a theoretical instrument and an algorithm to calculate similarity between benchmark tasks . they use relative performance of the "students" on a given task to determine similarity . |
| Outcome: | The proposed model reduces the number of evaluation tasks while maintaining high validation quality. |
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs. |
| Approach: | They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. |
| Outcome: | The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks. |
DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) require computational resources for fine-tuning. |
| Approach: | They propose a framework that optimizes rank allocation via two stages . they propose an initial pruning stage and a progressive pruning stage . |
| Outcome: | The proposed framework outperforms existing PEFT baselines on GLUE and instruction-following tasks while reducing training time and trainable parameters by over 80%. |