PLMS

UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation (2024.emnlp-main)

Challenge:	Existing pre-trained language models are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios.
Approach:	They propose a knowledge-injected pre-trained language model that can be transferred to both natural language understanding and generation tasks.
Outcome:	The proposed model significantly outperforms baselines across the board in e-commerce scenarios.

Copied to clipboard

Juhwan Choi, Yeonghwa Kim, Seunguk Yu, JungMin Yun, YoungBin Kim

Information Retrieval

Challenge:	Pre-trained language models have limited applicability for inference due to their large parameter size and limited generalization ability.
Approach:	They propose a method that generates a dataset regardless of the target domain . this allows for generalization of the tiny task model to any domain that shares the label space .
Outcome:	The proposed method achieves generalizability across domains while using a parameter set that is orders of magnitude smaller than PLMs.

Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning (2024.naacl-srw)

Copied to clipboard

Shih-Cheng Huang, Shih-Heng Wang, Min-Han Shih, Saurav Sahay, Hung-yi Lee

CharBERT: Character-aware Pre-trained Language Model (2020.coling-main)

Challenge:	Parameter-efficient (PE) methods for adapting pre-trained language models to downstream tasks are still lacking in many cases.
Approach:	They propose a general PE priming framework to enhance few-shot adaptation and generalization ability of PE methods.
Outcome:	The proposed framework reveals that the best priming strategy facilitates adaptation to target tasks.

Copied to clipboard

Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, Guoping Hu

AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level (2022.acl-long)

Challenge:	Pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations . but these methods split a word into subword units and make it incomplete and fragile .
Approach:	They propose a character-aware pre-trained language model to tackle OOV problems . they construct contextual word embedding for each token from sequential character representations .
Outcome:	The proposed model improves on the existing models on multiple NLP benchmarks.

Copied to clipboard

Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, Reut Tsarfaty

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese (2023.acl-srw)

Challenge:	a recent study shows that large pre-trained language models are not sufficient for Hebrew.
Approach:	They propose a large pre-trained language model for Hebrew that recovers morphological segments encoded in contextualized embedding vectors.
Outcome:	The proposed model obtains state-of-the-art on all tasks beyond contemporary Hebrew baselines.

Copied to clipboard

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, Yasuhiro Sogawa

Injecting Wiktionary to improve token-level contextual representations using contrastive learning (2024.eacl-short)

Challenge:	Existing studies on scriptio continua languages lack comprehensiveness of tokenizers . authors use Byte-Pair-Encoding or Unigram instead of WordPiece for subword tokenizer .
Approach:	They investigate the effect of tokenizers on the downstream performance of pretrained language models in scriptio continua languages where no explicit spaces exist between words.
Outcome:	The proposed tokenizers perform better on a wide range of tasks compared with other tokenizer methods . the results show that each task has an optimal morphological analyzer .

Copied to clipboard

Anna Mosolova, Marie Candito, Carlos Ramisch

Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards (2022.findings-emnlp)

Challenge:	lexical semantics tasks require contextual word embeddings that are not blind to context, despite the fact that vectors of the same meaning are too different.
Approach:	They propose to fine-tune pre-trained language models by using automatically self-augmented examples to target contextual word embeddings.
Outcome:	The proposed method achieves significant improvements on the original WiC test set and in two new tests.

Copied to clipboard

Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

OpenPrompt: An Open-source Framework for Prompt-learning (2022.acl-demo)

Challenge:	Existing work does not take full advantage of over-parameterized characteristics of large pre-trained language models.
Approach:	They propose a method that uses frozen "thinned" networks to obtain a mixture of rewards and advance the derivative-free prompt learning.
Outcome:	The proposed method outperforms previous gradient-free prompt learning methods and achieves parity with gradient-based counterparts on seven language understanding benchmarks under few-shot settings.

Copied to clipboard

Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, Maosong Sun

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2021.tacl-1)

Challenge:	Prompt-learning is a new paradigm in natural language processing, adapting pre-trained language models to cloze-style prediction, autoregressive modeling, or sequence to sequence generation.
Approach:	They propose a framework for prompt-learning that integrates pre-trained language models with a unified framework.
Outcome:	The proposed framework is easy to use and flexible enough to integrate with other frameworks.

Copied to clipboard

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, Jian Tang

REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training (2021.findings-acl)

Challenge:	Existing language representation models (PLMs) cannot capture factual knowledge from text.
Approach:	They propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation which integrates factual knowledge into PLMs and produces effective text-enhanced KE with the strong PLM.
Outcome:	The proposed model improves on existing pre-trained language representation models and improves their performance on various NLP tasks.

Copied to clipboard

Fangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji, Feng-Lin Li, Liqiang Nie

CULG: Commercial Universal Language Generation (2022.naacl-industry)

Challenge:	Pre-trained language models have achieved great success on Machine Reading Comprehension (MRC) however, the poor support in evidence extraction hinders them from further advancing MRC.
Approach:	They propose a REtrieval-based pre-training approach that strengthens evidence extraction during pre-training by inherited downstream MRC tasks.
Outcome:	The proposed approach strengthens evidence extraction during pre-training, which is further inherited by downstream tasks.

Copied to clipboard

Haonan Li, Yameng Huang, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise? (2023.starsem-1)

Challenge:	Pre-trained language models have improved performance for many NLP tasks in finance and healthcare.
Approach:	They propose a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages.
Outcome:	The proposed model outperforms other models on commercial generation tasks and on other markets, languages, and tasks.

Copied to clipboard

Xinzhe Li, Ming Liu, Shang Gao

Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding (2025.acl-srw)

Challenge:	Existing studies have shown that Pretrained Language Models (PLMs) perform poorly under noise due to subword segmentation.
Approach:	They propose a framework for subword segmentation that provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs.
Outcome:	The proposed framework provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs.

Copied to clipboard

Qi Feng, Yihong Liu, Hinrich Schuetze

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance (2025.naacl-industry)

Challenge:	Existing curriculum learning approaches rely on manually defined difficulty metrics which may not accurately reflect the model’s own perspective.
Approach:	They propose a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) they evaluate four datasets covering binary and multi-class classification tasks.
Outcome:	The proposed model leads to faster convergence and improved performance compared to standard random sampling.

Copied to clipboard

Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee

Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models (2021.acl-short)

Challenge:	Pretrained language models (PLMs) have revolutionized NLP but amplify linguistic inequities in multilingual applications.
Approach:	They evaluate pretrained language models including Mistral, TowerInstruct, OpenHathi, Tamil-Llama, and Kan-Lama across eight languages spanning high-resource and low-resourced settings.
Outcome:	The proposed models fail to bridge linguistic divides and are inefficient when compared to other models.

Copied to clipboard

Mingyue Han, Yinglin Wang

Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds (2022.naacl-main)

Challenge:	Pretrained language models (PLMs) achieve surprising performance on the Choice of Plausible Alternatives (COPA) task.
Approach:	They propose to add a regularization loss to the existing COPA models to mitigate the problem of semantic similarity bias by adding a normalization loss.
Outcome:	The proposed model improves generalization ability and performs better on a challenging dataset, BCOPA-CE, which has unbiased token distribution and is more difficult for models to distinguish cause and effect.

Copied to clipboard

Yu Zhang, Yu Meng, Xuan Wang, Sheng Wang, Jiawei Han

An Improved Baseline for Sentence-level Relation Extraction (2022.aacl-short)

Challenge:	Existing topic models adopt a fully unsupervised setting and their discovered topics may not reflect user preferences well due to their unsupervised nature.
Approach:	They propose a framework that allows out-of-vocabulary seeds to be used to find latent topics from text corpora.
Outcome:	The proposed framework can find topics that are never seen in the corpus and can benefit from the general knowledge of pre-trained language models.

Copied to clipboard

Wenxuan Zhou, Muhao Chen

Prompting for Multimodal Hateful Meme Classification (2022.emnlp-main)

Challenge:	Sentence-level relation extraction (RE) aims at identifying the relationship between two entities in a sentence.
Approach:	They propose to improve sentence-level relation extraction by adding entity representations with typed markers to the model.
Outcome:	The proposed model outperforms existing methods on entity representation and noisy labels on TACRED dataset.

Copied to clipboard

Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, Jing Jiang

BMInf: An Efficient Toolkit for Big Model Inference and Tuning (2022.acl-demo)

Challenge:	Hateful meme classification requires complex reasoning and contextual background knowledge.
Approach:	They propose a simple yet effective prompt-based model that prompts pre-trained language models for hateful meme classification.
Outcome:	The proposed model outperforms state-of-the-art models on hateful meme classification task.

Copied to clipboard

Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie Zhou, Jun Zhang, Jia Chao, Maosong Sun

Green & Sustainable NLP

Challenge:	Recent years, pre-trained language models (PLMs) have achieved promising results on various NLP tasks.
Approach:	They propose an open-source toolkit for big model inference and tuning which can support big model tuning at extremely low computation cost.
Outcome:	The proposed toolkit can support big model inference and tuning at extremely low computation cost.

PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting (2022.findings-naacl)

Copied to clipboard

Zhen Zhang, Wei Zhu, Jinfan Zhang, Peng Wang, Rize Jin, Tae-Sun Chung

Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models (2022.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) are the state-of-the-art (SOTA) models for natural language processing (NLP).
Approach:	They propose a patient and confident early exiting BERT (PCEE-BERT) that can work with different PLMs and popular model compression methods.
Outcome:	The proposed method outperforms existing models on the GLUE benchmarks and achieves different speed-up ratios.

Copied to clipboard

Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, Xu Sun

MVP-BERT: Multi-Vocab Pre-training for Chinese BERT (2021.acl-srw)

Challenge:	Existing methods for defending NLP models against backdoors have ignored the clean weights of PLMs.
Approach:	They exploit pre-trained weights to mitigate backdoors in fine-tuned NLP models . they use a fine-mixing technique and an Embedding Purification technique to do the same .
Outcome:	The proposed method outperforms baseline mitigation methods on three single-sentence sentiment classification tasks and two sentence-pair classification tasks.

Copied to clipboard

Wei Zhu

MedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries (2023.acl-srw)

Challenge:	Pre-trained language models (PLMs) improve performance of Chinese NLP tasks, but the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT (CITATION), which is based on Chinese characters (chars).
Approach:	They propose two versions of multi-vocab pre-training (MVP) to improve Chinese PLMs' expressiveness.
Outcome:	The proposed model outperforms the recent AMBERT (CITATION) model on span-level tasks and is more robust against adversarial attacks.

Copied to clipboard

Yang Cui, Lifeng Han, Goran Nenadic

Sub-Character Tokenization for Chinese Pretrained Language Models (2023.tacl-1)

Challenge:	Clinical texts contain important temporal information, such as medication start and end dates, appointment dates, and diagnosis dates.
Approach:	They propose to use prompt-based learning and fine-tuning to classify temporal relations between treatments and hospitalisation periods in discharge summaries.
Outcome:	The proposed method identifies whether a treatment was administered between the time of admission and discharge from the hospital.

Copied to clipboard

Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, Maosong Sun

Positive Text Reframing under Multi-strategy Optimization (2025.coling-main)

Challenge:	Existing tokenization methods for Chinese PLMs treat each character as an indivisible token, but ignore the unique feature of the writing system where additional linguistic information exists below the character level.
Approach:	They propose to encode Chinese characters into short sequences and construct Chinese vocabulary based on the encoded text.
Outcome:	The proposed tokenizers can tokenize inputs into much shorter sequences, improving computational efficiency.

Copied to clipboard

Shutong Jia, Biwei Cao, Qingqing Gao, Jiuxin Cao, Bo Liu

SEED: Semantic Knowledge Transfer for Language Model Adaptation to Materials Science (2024.emnlp-industry)

Challenge:	Existing positive reframing models can be fine-tuned to achieve acceptable results, but generating fluent, diverse text remains a challenge.
Approach:	They propose a positive reframing sentiment reward and content preservation reward framework . they propose re-ranking methods that optimize for style and consistency .
Outcome:	The proposed framework improves on unconstrained and controlled positive reframing tasks.

Copied to clipboard

Yeachan Kim, Jun-Hyung Park, SungHo Kim, Juhyeong Park, Sangyun Kim, SangKeun Lee

Contextualization Distillation from Large Language Model for Knowledge Graph Completion (2024.findings-eacl)

Challenge:	Existing methods to adapt pre-trained language models to materials science rely on frequency information from limited downstream datasets.
Approach:	They propose a vocabulary expansion method to adapt pre-trained language models to materials science by incorporating latent materials knowledge of lightweight embeddings into PLMs.
Outcome:	The proposed method mitigates the limitations of existing adaptation methods and can be used in materials science.

Copied to clipboard

Dawei Li, Zhen Tan, Tianlong Chen, Huan Liu

BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification (2023.findings-emnlp)

Challenge:	Existing knowledge graph completion models lack textual information, which limits their performance . a plug-in-and-play approach is needed to train small models in descriptive context .
Approach:	They propose a plug-in-and-play approach to knowledge graph completion that prompts LLMs to generate descriptive context.
Outcome:	The proposed method improves performance on Wikipedia articles and synset definitions.

Copied to clipboard

Dmitri Roussinov, Serge Sharoff

Information Retrieval

Challenge:	Pretrained language models have improved performance of text classification tasks, but they still suffer from spurious domain-specific clues.
Approach:	They propose a method to augment pretrained language models by generating texts in any desired genre and on any desired topic.
Outcome:	The proposed method improves on genre classification tasks while showing no improvement for other topics.

A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifiers (2024.emnlp-main)

Copied to clipboard

Valentin Barriere, Sebastian Cifuentes

Text Classification

Challenge:	Recent research shows that named entities influence PLMs in many applications.
Approach:	They propose a method to quantify biases associated with named entities from various countries using Twitter data instead of templates or specific datasets.
Outcome:	The proposed method shows positive biases related to the language spoken in a country across all classifiers.

Pruning Pre-trained Language Models Without Fine-Tuning (2023.acl-long)

Copied to clipboard

Ting Jiang, Deqing Wang, Fuzhen Zhuang, Ruobing Xie, Feng Xia

Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)

Challenge:	Existing methods to prune Pre-trained Language Models (PLMs) are overparameterized and require fine-tuning.
Approach:	They propose a pruning method that uses first-order pruning to prune PLMs while fine-tuning the remaining weights.
Outcome:	The proposed method outperforms first-order pruning and zero-order methods at sparsity levels.

Copied to clipboard

Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao

MAFIA: Multi-Adapter Fused Inclusive Language Models (2024.eacl-long)

Challenge:	Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks.
Approach:	They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks.
Outcome:	Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks.

Copied to clipboard

Prachi Jain, Ashutosh Sathe, Varun Gumma, Kabir Ahuja, Sunayana Sitaram

Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models (2024.findings-emnlp)

Challenge:	Pretrained Language Models (PLMs) are widely used in NLP for various tasks.
Approach:	They propose to modularly debias a pre-trained language model across multiple bias dimensions using structured knowledge and a large generative model.
Outcome:	The proposed model is able to debias a pre-trained language model across multiple bias dimensions in a semi-automated way.

Copied to clipboard

Vladimir Araujo, Marie-Francine Moens, Tinne Tuytelaars

Developing Prefix-Tuning Models for Hierarchical Text Classification (2022.emnlp-industry)

Challenge:	Recent work shows that PEFT methods can be competitive with, or even superior to, full fine-tuning of PLMs.
Approach:	They propose a method that isolates the training of new PEFT modules to ensure their task specialization and learns to compose them by training a network of routers that leverages a small memory containing examples of previously seen tasks.
Outcome:	The proposed method improves generalization and performance in two CL setups.

Copied to clipboard

Lei Chen, Houwei Chou, Xiaodan Zhu

Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models (2021.findings-emnlp)

Challenge:	Hierarchical text classification (HTC) is a key task in many industrial applications. Pre-trained Language Models (PLMs) have become dominant for most natural language processing (NLP) tasks.
Approach:	They investigate how prefix tuning can improve hierarchical text classification . prefix-tuning model only needs less than 1% of parameters to achieve performance .
Outcome:	The proposed model can achieve comparable performance to regular full fine-tuning.

Copied to clipboard

Taeuk Kim, Bowen Li, Sang-goo Lee

Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)

Challenge:	Existing methods for extracting complete (binary) parses from pre-trained language models are expensive and time-consuming.
Approach:	They propose a chart-based method and an effective top-K ensemble technique to extractbinary parses from PLMs.
Outcome:	The proposed method can induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, and is robust to cross-lingual transfer.

Copied to clipboard

Bowen Li, Taeuk Kim, Reinald Kim Amplayo, Frank Keller

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)

Challenge:	Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited.
Approach:	They propose an unsupervised method that extracts constituency trees from PLM attention heads.
Outcome:	The proposed method outperforms existing approaches if no development set is present.

Copied to clipboard

Valentin Hofmann, Hinrich Schuetze, Janet Pierrehumbert

A Table-to-Text Framework with Heterogeneous Multidominance Attention and Self-Evaluated Multi-Pass Deliberation (2023.findings-emnlp)

Challenge:	a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed .
Approach:	They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure .
Outcome:	The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks.

Copied to clipboard

Xi Chen, Xinjiang Lu, Haoran Xin, Wenjun Peng, Haoyang Duan, Feihu Jiang, Jingbo Zhou, Hui Xiong

Structured Data in NLP

Challenge:	Table-to-text works have been widely applied in different domains, such as weather forecast and financial report generation.
Approach:	They propose a table-to-text approach on top of Self-evaluated multi-pass Generation and Heterogenous Multidominance Attention to explore the hierarchical structure.
Outcome:	The proposed method outperforms several SOTA methods quantitatively and qualitatively on three public datasets.

Abstractive Document Summarization with Summary-length Prediction (2023.findings-eacl)

Copied to clipboard

Jingun Kwon, Hidetaka Kamigaito, Manabu Okumura

BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting (2023.acl-industry)

Challenge:	Existing abstractive summarization models do not consider summarizing-specific information such as the target summary length.
Approach:	They propose a method for enabling a model to understand summarization-specific information by predicting the summary length in the encoder and generating a summary of the predicted length in fine-tuning.
Outcome:	The proposed method improves ROUGE scores on the WikiHow, NYT, and CNN/DM datasets.

Copied to clipboard

Wei Zhu, Peng Wang, Yuan Ni, Guotong Xie, Xiaoling Wang

Assessing Combinational Generalization of Language Models in Biased Scenarios (2022.aacl-short)

Challenge:	Recent years have witnessed the rise of many pre-trained language models (PLMs) such as GPT (Radford et al., 2019) and XLNet (Yang e.t al, 2019).
Approach:	They propose a framework which consists of two off-the-shelf methods for improving PLMs’ early exiting.
Outcome:	The proposed method can reduce the average latency of pre-trained language models and work with other inference speed-up methods like model pruning.

Copied to clipboard

Yanbo Fang, Zuohui Fu, Xin Dong, Yongfeng Zhang, Gerard de Melo

Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model (2024.emnlp-industry)

Challenge:	Existing work focuses on assessing in-domain knowledge, but shedding light on what pre-trained Language Models learn is important.
Approach:	They propose a method to assess a PLM's generalization capacity in biased scenarios by combining component combinations where it could be easy for the PLMs to learn shortcuts from the training corpus.
Outcome:	The proposed model can overcome distribution shifts in the training corpus and with sufficient data.

Copied to clipboard

Subhadip Nandi, Neeraj Agrawal

LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework (2022.findings-naacl)

Challenge:	Existing approaches to named entity recognition are domain specific and require a domain specific architecture.
Approach:	They propose a retrieval augmented large language model for Named Entity Recognition . the model uses word-embedding over sentence-level embedding to fine tune .
Outcome:	The proposed model outperforms existing models on the CrossNER dataset.

Copied to clipboard

Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, Hinrich Schuetze

Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters (2023.eacl-main)

Challenge:	Recent work shows that large-scale pretrained language models (PLMs) are effective few-shot learners.
Approach:	They propose a method that treats few-shotlearners as crowdsourcing workers . they propose to use these workers to train models that solve a task well .
Outcome:	The proposed approach treats few-shotlearners as crowdsourcing workers . the resulting annotations can be utilized to train models that solve the task well .

Copied to clipboard

Sebastien Montella, Alexis Nasr, Johannes Heinecke, Frederic Bechet, Lina M. Rojas Barahona

Probing Critical Learning Dynamics of PLMs for Hate Speech Detection (2024.findings-eacl)

Challenge:	Recent approaches to text generation from Abstract Meaning Representation (AMR) have been based on neural-centered encoderdecoder architectures.
Approach:	They propose a structure-aware adapter which injects the input graph connectivity within PLMs using Graph Neural Networks.
Outcome:	The proposed adapter is robust to a variety of approaches and can be used to generate Graph-to-Text representations.

Copied to clipboard

Sarah Masud, Mohammad Aflah Khan, Vikram Goyal, Md Shad Akhtar, Tanmoy Chakraborty

Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses? (2023.eacl-main)

Challenge:	Existing studies on pretrained language models (PLMs) for hate speech detection have not investigated how their performance is affected by pretraining and finetuning.
Approach:	They propose to compare pretrained language models, evaluate their seed robustness, finetuning settings, and the impact of pretraining data collection time.
Outcome:	The proposed models show that they are more robust than other models and that they have a better chance of performing better than domain-specific models.

Copied to clipboard

Avi Shmidman, Cheyn Shmuel Shmidman, Dan Bareket, Moshe Koppel, Reut Tsarfaty

Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Challenge:	Semitic morphologically-rich languages are characterized by extreme word ambiguity . many of the words are homographs with multiple possible analyses .
Approach:	They evaluate existing models for Hebrew homographs using word-piece embeddings . they find they are more effective when the number of word-part splits is limited .
Outcome:	The proposed models outperform non-contextualized embeddings on Hebrew homograph challenge sets.

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

Probing Power by Prompting: Harnessing Pre-trained Language Models for Power Connotation Framing (2023.eacl-main)

Challenge:	In this paper, we examine whether pretrained language models are consistent with factual knowledge.
Approach:	They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue .
Outcome:	The proposed model improves consistency and shows that it is effective.

Copied to clipboard

Shima Khanehzar, Trevor Cohn, Gosia Mikolajczak, Lea Frermann

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (2024.findings-naacl)

Challenge:	Using pre-trained language models, we investigated whether word choices can encode subtle connotative information about power differentials between involved entities.
Approach:	They propose a framework to disentangle connotation frames implied by the predicate from its arguments and the sentence structure and to quantify predicates.
Outcome:	The proposed framework improves power connotation prediction accuracy by fine-tuning pre-trained language models.

Copied to clipboard

Yihong Liu, Peiqin Lin, Mingyang Wang, Hinrich Schuetze

Autoregressive Structured Prediction with Language Models (2022.findings-emnlp)

Challenge:	Existing methods to pretrain multilingual models are limited by the number of embedding parameters and the complexity of the model.
Approach:	They propose a framework that initializes the embeddings of unseen subwords and can adapt a model to multiple languages.
Outcome:	The proposed framework can adapt a pre-trained model to multiple languages efficiently and effectively.

Copied to clipboard

Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, Mrinmaya Sachan

Semantic Frame Induction from a Real-World Corpus (2025.acl-srw)

Challenge:	Recent years have seen a paradigm shift in NLP towards using pretrained language models for a wide range of tasks.
Approach:	They propose to model structures as sequences of actions in autoregressive manner with PLMs . their approach allows in-structure dependencies to be learned without any loss .
Outcome:	The proposed approach achieves state-of-the-art on all structured prediction tasks.

Copied to clipboard

Shogo Tsujimoto, Kosuke Yamada, Ryohei Sasano

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better (2022.acl-short)

Challenge:	Existing studies on semantic frame induction have demonstrated that pre-trained language models (PLMs) have led to more accurate results.
Approach:	They conduct semantic frame induction using the Colossal Clean Crawled Corpus and assess the applicability of existing frame inducing methods to real-world data.
Outcome:	The proposed methods outperform existing methods on real-world data and can induce frames corresponding to novel concepts.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang

Adaptive Structure Induction for Aspect-based Sentiment Analysis with Spectral Perspective (2023.findings-emnlp)

Challenge:	Existing methods for finetuning pretrained language models (PLMs) have risks in overfitting the pretraining tasks and data, which may lead to suboptimal performance.
Approach:	They propose a method which adds noise to parameters of PLMs before fine-tuning.
Outcome:	The proposed method can be used on GLUE English and XTREME multilingual benchmarks.

Copied to clipboard

Hao Niu, Yun Xiong, Xiaosu Wang, Wenjing Yu, Yao Zhang, Zhonglei Guo

Sentiment Analysis

Challenge:	incorporating structure information can enhance the performance of aspect-based sentiment analysis.
Approach:	They propose to use pre-trained language models to induct latent structures from a spectrum perspective.
Outcome:	The proposed model shortens Aspects-sentiment Distance and improves structure induction ability.

SelfMix: Robust Learning against Textual Label Noise with Self-Mixup Training (2022.coling-1)

Copied to clipboard

Dan Qiao, Chenchen Dai, Yuyang Ding, Juntao Li, Qiang Chen, Wenliang Chen, Min Zhang

RGL: A Simple yet Effective Relation Graph Augmented Prompt-based Tuning Approach for Few-Shot Learning (2022.findings-naacl)

Challenge:	Existing methods to handle label noise in text classification tasks are limited to visual data.
Approach:	They propose a method to handle label noise in text classification tasks using a Gaussian Mixture Model.
Outcome:	The proposed method outperforms baselines on three types of text classification tasks on visual and textual data.

Copied to clipboard

Yaqing Wang, Xin Tian, Haoyi Xiong, Yueyang Li, Zeyu Chen, Sheng Guo, Dejing Dou

Analyzing Homonymy Disambiguation Capabilities of Pretrained Language Models (2024.lrec-main)

Challenge:	Pre-trained language models (PLMs) are a good starting point for downstream applications, but it is difficult to generalize them to new tasks given a few labeled samples.
Approach:	They propose to use Relation Graph augmented learning to improve the performance of few-shot natural language understanding tasks by rewriting the input sequence into a cloze question with masks.
Outcome:	Extensive experiments show that Relation Graph augmented learning (RGL) improves performance of prompt-based tuning strategies.

Copied to clipboard

Lorenzo Proietti, Stefano Perrella, Simone Tedeschi, Giulia Vulpis, Leonardo Lavalle, Andrea Sanchietti, Andrea Ferrari, Roberto Navigli

Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022.findings-naacl)

Challenge:	Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) but current pretrained language models lack the granularity to perform disambiguation .
Approach:	They propose a large-scale resource that leverages homonymy relations to cluster WordNet senses and train Homonymy Disambiguation systems.
Outcome:	The proposed model can distinguish homonyms with up to 95% accuracy even without fine-tuning the underlying PLM.

Copied to clipboard

Vaibhav Varshney, Mayur Patidar, Rajat Kumar, Lovekesh Vig, Gautam Shroff

RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Models (2024.findings-acl)

Challenge:	Existing methods to identify all possible user intents at design time are expensive and require storage of past data.
Approach:	They propose to continually train an intent detector on new intents while maintaining performance on prior intents.
Outcome:	The proposed method outperforms exemplar replay-based approaches on lifelong intent detection tasks and achieves state-of-the-art on four public datasets.

Copied to clipboard

Saeed Najafi, Alona Fyshe

PLM-based World Models for Text-based Games (2022.emnlp-main)

Challenge:	Pre-trained Language Models (PLMs) can be fine-tuned for downstream text processing tasks.
Approach:	They propose to use paraphrases to enrich the input text of a few-shot model with a Maximum-Marginal Likelihood objective to improve performance.
Outcome:	The proposed methods improve performance beyond what can be achieved with parameter-efficient fine-tuning alone.

Copied to clipboard

Minsoo Kim, Yeonjoon Jung, Dohyeon Lee, Seung-won Hwang

Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (2024.eacl-long)

Challenge:	a new study shows that pre-trained world models provide a strong base for world models . worldformer is a text-based game environment that can be used to learn world models in text-driven games.
Approach:	They propose to use pre-trained language models to build world models in text-based game environments.
Outcome:	The proposed model outperforms state-of-the-art model-free algorithms in Atari games while retaining sample efficiency.

Copied to clipboard

Haoqiang Kang, Terra Blevins, Luke Zettlemoyer

How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts (2023.findings-emnlp)

Challenge:	Pretrained language models learn cross-lingual knowledge and perform well on diverse tasks when finetuned.
Approach:	They propose a zero-shot prompting approach that captures cross-lingual word sense with a contextual prompt.
Outcome:	The proposed approach outperforms baselines on recall in many evaluation languages without additional training or finetuning.

Copied to clipboard

Tharindu Kumarage, Paras Sheth, Raha Moraffah, Joshua Garland, Huan Liu

Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models (2023.acl-long)

Challenge:	Existing methods to detect AI-generated text are inadequate, causing misuse of the text.
Approach:	They propose a universal evasive prompt framework that can prompt any PLM to generate “human-like” text that can mislead detectors.
Outcome:	The proposed approach can prompt any PLM to generate “human-like” text that can mislead detectors.

Copied to clipboard

Zhong Zhang, Bang Liu, Junming Shao

Language Model Pre-Training with Sparse Latent Typing (2022.emnlp-main)

Challenge:	Pre-trained language models are overly parameterized and have significant redundancy . recent studies show that PLMs are highly over-parameterized and robust to pruning .
Approach:	They propose to re-parameter and fine-tune pre-trained language models from a new perspective: Discovery of intrinsic task-specific subspace.
Outcome:	The proposed model can be fine-tuned in the subspace with a small number of free parameters.

Copied to clipboard

Liliang Ren, Zixuan Zhang, Han Wang, Clare Voss, ChengXiang Zhai, Heng Ji

Rethinking Data Augmentation in Text-to-text Paradigm (2022.coling-1)

Challenge:	Modern large-scale Pre-trained Language Models focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences.
Approach:	They propose a new pre-training objective that enables the model to learn latent types . the objective allows the model a self-supervised way to extract sentence-level keywords .
Outcome:	The proposed model learns interpretable latent type categories without external knowledge and improves downstream tasks.

Copied to clipboard

Yanan Chen, Yang Liu

CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)

Challenge:	Existing approaches to augment training data are limited or marginal, or even diminishing or adverse especially given original training corpus is relatively sufficient or the backbone classifiers are PLM based.
Approach:	They propose to integrate text-to-text language models and construct a new two-phase framework for augmentation using two novel schemes.
Outcome:	The proposed framework synthesizes new samples benefiting from the knowledge learned from pre-trained language models on two public classification datasets and shows remarkable gains.

Copied to clipboard

Divya Jyoti Bajpai, Manjesh Hanawal

TagDebias: Entity and Concept Tagging for Social Bias Mitigation in Pretrained Language Models (2024.findings-naacl)

Challenge:	Pre-trained Language Models suffer in inference latency due to their large size.
Approach:	They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers.
Outcome:	The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy.

Copied to clipboard

Mehrnaz Moslemi, Amal Zouaq

AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models (2022.naacl-main)

Challenge:	Existing methods to mitigate social bias in pre-trained language models are limited . researchers have found that these models inherit substantial social biases in their pre-training data .
Approach:	They propose a method which proposes debiasing a dataset using type tags and fine-tunes PLMs on this debiased dataset.
Outcome:	The proposed method improves bias scores on a ranking task . it is based on analyzing type tags and fine-tuning pre-trained models .

Copied to clipboard

Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, Chao Zhang

Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain (2023.eacl-main)

Challenge:	Existing methods for fine-tuning pre-trained language models ignore the potential of unlabeled data.
Approach:	They propose a framework that allows users to unleash the power of unlabeled data via self-training.
Outcome:	The proposed framework outperforms active learning and self-training baselines and improves the label efficiency of PLM fine-tuning by 56.2% on average.

Copied to clipboard

Annerose Eichel, Helena Schlipf, Sabine Schulte im Walde

In-Context Analogical Reasoning with Pre-Trained Language Models (2023.acl-long)

Challenge:	a novel approach to learn domain-specific plausible materials for components in the vehicle repair domain is proposed . connecting a symptom to an underlying cause is a crucial building block for natural language understanding across domains.
Approach:	They propose a method to aggregate salient predictions from a set of cloze task style templates and use a Wikipedia corpus to augment the model.
Outcome:	The proposed approach outperforms a traditional pattern-based approach by exploiting the compositionality assumption in a cloze task style setting.

Copied to clipboard

Xiaoyang Hu, Shane Storks, Richard Lewis, Joyce Chai

BabyLM’s First Constructions: Causal interventions provide a signal of learning (2025.emnlp-main)

Challenge:	Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences.
Approach:	They apply large pre-trained language models to visual Raven’s Progressive Matrices (RPM) and use language-based abstractions to support analogy in AI systems.
Outcome:	The proposed language-based abstractions outperform human models on Raven’s Progressive Matrices and supervised vision-based methods.

Copied to clipboard

Joshua Rozner, Leonie Weissweiler, Cory Shain

How Much Do Encoder Models Know About Word Senses? (2025.acl-long)

Challenge:	Recent work shows sensitivity to constructions in pretrained language models, but their relevance to human language learning is doubted.
Approach:	They use construction grammars to demonstrate sensitivity to constructions in pretrained language models.
Outcome:	The proposed models learn diverse constructions even hard cases that are superficially indistinguishable.

Copied to clipboard

Simone Teglia, Simone Tedeschi, Roberto Navigli

SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations (2024.acl-long)

Challenge:	Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) however, how well these models inherently disambiguate word senses remains uncertain.
Approach:	They evaluate several encoder-only PLMs across WordNet and ODE sense inventories to evaluate their ability to separate word senses without any task-specific fine-tuning.
Outcome:	The proposed model outperforms output layer on WordNet and ODE sense inventories by 15 percentage points.

Copied to clipboard

Jesus Solano, Mardhiyah Sanni, Oana-Maria Camburu, Pasquale Minervini

A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022.coling-1)

Challenge:	Models that generate natural language explanations (NLEs) for their predictions often require large datasets of human-written NLEs at training time, which can be expensive and time-consuming to collect.
Approach:	They propose a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs.
Outcome:	The proposed approach compares sparse few-shot fine-tuning with existing parametric fine- tuning techniques on three sizes of the T5 language model and four datasets and produces competitive results for both task performance and NLE quality.

Copied to clipboard

Pranav Narayanan Venkit, Mukund Srinath, Shomir Wilson

Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers (2023.findings-eacl)

Challenge:	Pretrained language models exhibit sociodemographic biases, such as against gender and race, raising concerns of downstream biase in language technologies.
Approach:	They propose to use word embedding-based and transformer-based PLMs to test for the presence of biases against people with disabilities (PWDs)
Outcome:	The proposed models favor ableist language, despite their sociodemographic biases against race and gender.

Copied to clipboard

Chia-Chien Hung, Anne Lauscher, Dirk Hovy, Simone Paolo Ponzetto, Goran Glavaš

DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization (2025.findings-naacl)

Challenge:	Existing studies show that incorporating demographic factors in language representations improves performance on downstream NLP tasks.
Approach:	They use continuous language modeling and dynamic multi-task learning to adapt pre-trained Transformers to incorporate demographic information into their representations.
Outcome:	The proposed model shows that the results are consistent with previous studies.

Copied to clipboard

Haohan Yuan, Haopeng Zhang

Summarization

Challenge:	Abstractive summarization is a crucial task in natural language processing . current research focuses on summarizing specific types of documents . domain shifts between documents affect summarisation performance .
Approach:	They propose a hierarchical benchmark to capture fine-grained domain shifts in abstractive summarization.
Outcome:	The proposed benchmark measures the generalization capabilities of pre-trained language models and large language models in in-domain and cross-domain settings.

TempLM: Distilling Language Models into Template-Based Generators (2023.findings-acl)

Copied to clipboard

Tianyi Zhang, Mina Lee, Xiang Lisa Li, Ende Shen, Tatsunori Hashimoto

Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)

Challenge:	Pretrained language models (PLMs) have greatly improved text generation, but they have also been known to produce unfaithful or inappropriate content.
Approach:	They propose a pretrained language model that is a template-based generator and uses it to generate a text.
Outcome:	The proposed model is more faithful than the original PLM and more fluent than prior template systems.

Copied to clipboard

Nalin Kumar, Ondrej Dusek

Post-Training Dialogue Summarization using Pseudo-Paraphrasing (2022.findings-naacl)

Challenge:	Existing approaches to building monolingual models for low-resource languages require a full model tuning process.
Approach:	They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language.
Outcome:	The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language.

Copied to clipboard

Qi Jia, Yizhu Liu, Haifeng Tang, Kenny Zhu

ArT: All-round Thinker for Unsupervised Commonsense Question Answering (2022.coling-1)

Challenge:	Existing approaches to dialogue summarization use dialogue-specific features that require additional knowledge to recognize or make the models harder to tune.
Approach:	They propose to post-train pretrained language models to rephrase from dialogue to narratives and fine-tune them as usual.
Outcome:	The proposed approach outperforms existing models by summary quality and implementation costs.

Copied to clipboard

Jiawei Wang, Hai Zhao

Do Language Models Understand Measurements? (2022.findings-emnlp)

Challenge:	Existing work on commonsense QA requires labeled training data for its success . existing work relies on large-scale in-domain or out-of-domain labeles or fails to generate knowledge of high quality in a general way.
Approach:	They propose an approach to commonsense question-answering (QA) that takes association during knowledge generation.
Outcome:	The proposed model outperforms existing models on commonsense QA benchmarks.

Copied to clipboard

Sungjin Park, Seungwoo Ryu, Edward Choi

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding (2023.eacl-main)

Challenge:	Existing studies on numerical reasoning over text (NRoT) tests PLMs to understand numbers in contexts where numbers are an integral part of the context.
Approach:	They propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in probing tasks.
Outcome:	The proposed model distinguishes between numbers and units, which leads to significant improvement in probing tasks.

Copied to clipboard

Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Hassan Awadallah

Exploiting Abstract Meaning Representation for Open-Domain Question Answering (2023.findings-acl)

Challenge:	Recent studies have focused on compressing pre-trained language models (PLMs) however, few studies have examined the impact of compression on generalizability and robustness of compressed models for out-of-distribution data.
Approach:	They propose to use knowledge distillation and pruning to reduce model generalization and generalization on out-of-distribution data.
Outcome:	The proposed compression techniques overfit on shortcut samples and generalize poorly on hard ones.

Copied to clipboard

Cunxiang Wang, Zhikun Xu, Qipeng Guo, Xiangkun Hu, Xuefeng Bai, Zheng Zhang, Yue Zhang

GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models (2022.emnlp-main)

Challenge:	Existing work attempts to address these challenges using Pretrained Language Models (PLMs) but the diversity of surface form expressions can hinder the model’s ability to capture accurate correlations, especially when the context is lengthy and complex.
Approach:	They propose a method known as Graph-as-Token (GST) to incorporate AMRs into PLMs to assist the model in understanding complex semantic information.
Outcome:	The proposed method outperforms existing methods and significantly improves performance on both Natural Questions and TriviaQA.

Copied to clipboard

Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, Kai-Wei Chang

The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification (2023.eacl-main)

Challenge:	Recent work shows that Pre-trained Language Models store relational knowledge and utilize it for performing downstream tasks.
Approach:	They propose a benchmark dataset for probing the diversity of relational knowledge in multilingual PLMs.
Outcome:	The proposed dataset contains 3125 prompts in English, Chinese, Hindi, Persian, and Swahili . larger multilingual PLMs variants do not store geo-diverse concepts better than its smaller variant .

Copied to clipboard

Mohsen Mesgar, Thy Thy Tran, Goran Glavaš, Iryna Gurevych

How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis (2022.findings-acl)

Challenge:	Recent methods for task-oriented dialog (ToD) intent classification use pretrained language models . but lack of informative ablations prevents identification of factors that drive performance .
Approach:	They propose a framework to evaluate components of Few-Shot Intent Classification . they propose to combine cross-encoder architecture and episodic meta-learning .
Outcome:	The proposed framework evaluates cross-encoder architecture and episodic meta-learning . it also shows that splitting episodes into support and query sets outperforms non-episodic counterparts.

Copied to clipboard

Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, Qun Liu

A Self-verified Method for Exploring Simile Knowledge from Pre-trained Language Models (2024.lrec-main)

Challenge:	Recent studies show that pre-trained language models can fill in the missing factual words in cloze-style prompts such as ”Dante was born in [MASK]” .
Approach:	They propose to quantitatively measure and evaluate the word-level patterns that PLMs depend on to generate the missing factual words.
Outcome:	The proposed model fills in the missing factual words in cloze-style prompts by relying on effective clues or shortcut patterns.

Copied to clipboard

Longxuan Ma, Changxin Ke, Shuhan Zhou, Churui Sun, Wei-Nan Zhang, Ting Liu

Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) have succeeded in natural language processing because they learn generic knowledge from a large corpus.
Approach:	They propose a method that allows pre-trained language models to explore simile knowledge from PLMs . they enhance PLM models with a multi-level simile recognition task that evaluates similes aplenty .
Outcome:	The proposed method can explore more accurate simile knowledge for PLMs.

Copied to clipboard

Jaeyoung Choe, Keonwoong Noh, Nayeon Kim, Seyun Ahn, Woohwan Jung

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling (2022.naacl-main)

Challenge:	Existing financial PLMs are not pretrained on sufficiently diverse financial data, leading to subpar generalization performance.
Approach:	They propose to pretrain financial PLMs on financial corpus and train financial models on financial data.
Outcome:	The proposed financial language models outperform existing financial PLMs on financial tasks even for unseen corpus groups.

Copied to clipboard

Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Daxin Jiang

A Semantic Mention Graph Augmented Model for Document-Level Event Argument Extraction (2024.lrec-main)

Challenge:	Existing methods to train cross-lingual pre-trained language models have shown great success in cross-linguistic sequence labeling tasks.
Approach:	They propose a cross-lingual language informative span masking task to eliminate the objective gap between pre-training and fine-tuning stages.
Outcome:	The proposed method surpasses the state-of-the-art methods on multiple benchmarks even with limited pre-training data.

Copied to clipboard

Jian Zhang, Changlin Yang, Haiping Zhu, Qika Lin, Fangzhi Xu, Jun Liu

On Dataset Transferability in Active Learning for Transformers (2023.findings-acl)

Challenge:	Document-level Event Argument Extraction (DEAE) aims to identify arguments and their specific roles from unstructured document.
Approach:	They propose a document-prompt-based method for document-level event argument extraction that uses a semantic mention graph to capture relations between documents and prompts.
Outcome:	The proposed method surpasses baseline methods and achieves state-of-the-art performance on RAMS and WikiEvents datasets.

Copied to clipboard

Fran Jelenić, Josip Jukić, Nina Drobac, Jan Snajder

Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022.findings-emnlp)

Challenge:	Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning.
Approach:	They propose to query examples most beneficial for model learning by querying data points most informative for labeling.
Outcome:	The proposed method reduces labeling costs by querying the examples most beneficial for model learning.

Copied to clipboard

Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang

Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Bias (2023.findings-acl)

Challenge:	Pre-trained language models (PLMs) are prone to leaking personal information due to memorization, but the risk of specific personal information being extracted by attackers is low.
Approach:	They analyze whether large pre-trained language models are prone to leaking personal information due to memorization.
Outcome:	The proposed model is weak at association, so the risk of specific personal information being extracted by attackers is low.

Copied to clipboard

Zhiyuan Zhang, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, Xu Sun

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2022.acl-long)

Challenge:	Existing methods to fine-tune pre-trained language models (PLMs) are not safe, since the fine-uning process is invisible to the user.
Approach:	They propose a technique to study the dynamic process of fine-tuning for finding poisonous dimensions using diffusion theory.
Outcome:	The proposed approach can detect poisonous dimensions with abnormal dynamics, purify them and fine-tune them on a clean dataset.

Copied to clipboard

Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, Maosong Sun

Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022.findings-naacl)

Challenge:	Recent studies suggest that pre-trained language models have gained rich knowledge during pre-training.
Approach:	They propose to tune pre-trained language models with task-specific prompts to improve and stabilize prompttuning.
Outcome:	Extensive experiments on zero and few-shot text classification tasks show that prompt-tuning improves and stabilizes prompttun-ing.

Copied to clipboard

Siyu Ren, Kenny Zhu

Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022.emnlp-main)

Challenge:	Pretrained masked language models inherit a considerable amount of relational knowledge from the source corpora.
Approach:	They propose to specialize pretrained masked language models into relational models from the perspective of network pruning.
Outcome:	The proposed model can represent grounded commonsense relations at non-trivial sparsity while being generalizable . the proposed model improves on a wealth of NLP tasks, but we know little about how much knowledge it imparts .

Copied to clipboard

Boshi Wang, Xiang Deng, Huan Sun

Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models (2023.eacl-main)

Challenge:	Pre-trained language models (PLMs) internalize a great amount of knowledge, but have been shown incapable of recalling this knowledge to solve complex & multi-step reasoning tasks.
Approach:	They propose an iterative prompting framework which progressively elicits relevant knowledge from PLMs for multi-step inference.
Outcome:	The proposed prompting framework outperforms existing prompting methods on three datasets involving multi-step reasoning.

Copied to clipboard

Zdeněk Kasner, Ioannis Konstas, Ondrej Dusek

What do tokens know about their characters and how do they know it? (2022.naacl-main)

Challenge:	Pretrained language models (PLMs) for data-to-text generation produce inaccurate outputs if labels are ambiguous or incomplete, which is often the case in D2T datasets.
Approach:	They propose to use a dataset to descib a relation between two entities using relation labels to train pretrained language models.
Outcome:	The proposed models are robust to generalizing to out-of-domain domains on a dataset for descibing a relation between two entities.

Copied to clipboard

Ayush Kaushal, Kyle Mahowald

Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)

Challenge:	Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information.
Approach:	They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information .
Outcome:	The proposed models can encode character-level information and perform better on non-Latin alphabets.

Copied to clipboard

Philipp Dufter, Nora Kassner, Hinrich Schütze

SP3: Enhancing Structured Pruning via PCA Projection (2024.findings-acl)

Challenge:	Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes .
Approach:	They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set .
Outcome:	The results show that static embeddings perform better than PLMs when restricted to a candidate set .

Copied to clipboard

Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, Hong Chen

Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction (2024.acl-long)

Challenge:	Structured pruning is a widely used technique for reducing the size of pre-trained language models, but current methods overlook the potential of compressing the hidden dimension d in PLMs.
Approach:	They propose a structured pruning approach that projectes features into a space defined by principal components before masking the hidden dimension d in pre-trained language models.
Outcome:	Experiments on benchmarks show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy.

Copied to clipboard

Jianhao Chen, Haoyuan Ouyang, Junyang Ren, Wentao Ding, Wei Hu, Yuzhong Qu

Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues (2023.findings-eacl)

Challenge:	Recent research on temporal fact extraction fails to establish time-to-fact correspondences in complex sentences.
Approach:	They propose a timeline-based sentence decomposition strategy using large language models with in-context learning to extract temporal facts from natural language text.
Outcome:	The proposed method achieves state-of-the-art on a complex temporal fact extraction dataset.

Copied to clipboard

Chuyuan Li, Patrick Huber, Wen Xiao, Maxime Amblard, Chloe Braud, Giuseppe Carenini

Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models (2024.naacl-long)

Challenge:	Discourse processing suffers from data sparsity, especially for dialogues . a variety of discourse frameworks have been proposed to extract discourse information from dialogues.
Approach:	They propose unsupervised and semi-supervised methods to infer latent discourse structures for dialogues based on attention matrices from Pre-trained Language Models.
Outcome:	The proposed methods achieve encouraging results on the STAC corpus, with F1 scores of 57.2 and 59.3 for the unsupervised and semi-supervised methods, respectively.

Copied to clipboard

Stephan Linzbach, Dimitar Dimitrov, Laura Kallmeyer, Kilian Evang, Hajira Jabeen, Stefan Dietze

Knowledge Rumination for Pre-trained Language Models (2023.emnlp-main)

Challenge:	Pre-trained language models contain various kinds of knowledge.
Approach:	They designed a probe that allows comparison of 34 million distinct paraphrases that follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations.
Outcome:	Extensive knowledge retrieval experiments show that prompts following clausal syntax have several desirable properties in comparison to appositive syntax.

Copied to clipboard

Yunzhi Yao, Peng Wang, Shengyu Mao, Chuanqi Tan, Fei Huang, Huajun Chen, Ningyu Zhang

Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization (2023.acl-long)

Challenge:	Existing studies have shown that pre-trained language models lack the capacity to handle knowledge-intensive tasks alone.
Approach:	They propose a new paradigm to help pre-trained language models utilize latent knowledge without retrieving it from external corpus.
Outcome:	The proposed paradigm can be applied to pre-trained language models without retrieving external knowledge from the corpus.

Copied to clipboard

Ze-Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao, Ji-Rong Wen

COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models (2023.eacl-main)

Challenge:	Large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, outperforming small PLMs by a large margin.
Approach:	They propose to scale up parameters of pre-trained language models only during fine-tuning to benefit from over-parameterization.
Outcome:	The proposed approach can significantly boost the fine-tuning performance of small PLMs and even help small PDMs outperform 3 parameterized larger ones.

Copied to clipboard

Kanishka Misra, Julia Rayz, Allyson Ettinger

Progressive Class Semantic Matching for Semi-supervised Text Classification (2022.naacl-main)

Challenge:	Existing pre-trained language models (PLMs) lack robustness in demonstrating simple reasoning, despite having the prerequisite knowledge.
Approach:	They propose to test pre-trained language models' ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior.
Outcome:	The proposed model can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the base of nuanced knowledge representations.

Copied to clipboard

Haiming Xu, Lingqiao Liu, Ehsan Abbasnejad

Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)

Challenge:	Recent semi-supervised learning methods have achieved impressive performance . semi-controlled learning can be used to reduce the annotation cost of text classifiers .
Approach:	They propose a semi-supervised learning process that builds a standard K-way classifier and a matching network for the input text and the Class Semantic Representation (CSR).
Outcome:	The proposed method improves baselines and overall is more stable.

Copied to clipboard

Yuxin Jiang, Linhan Zhang, Wei Wang

Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023.emnlp-main)

Challenge:	Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings.
Approach:	They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning.
Outcome:	Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task.

Copied to clipboard

Sapan Shah, Sreedhar Reddy, Pushpak Bhattacharyya

Enhancing Lexical Relation Mining with Structured Sememe Knowledge (2026.acl-long)

Challenge:	a novel retrofitting method to induce emotion aspects into pre-trained language models is proposed . the models are computationally less expensive and open, but do not capture affective aspects of human communication well.
Approach:	They propose a retrofitting method to induce emotion aspects into pre-trained language models . they retrofit text fragments exhibiting similar emotions into pretrained networks .
Outcome:	The proposed method produces emotion-aware text representations for sentiment analysis and sarcasm detection tasks.

Copied to clipboard

Hansi Wang, Qiliang Liang, Yue Wang, Yang Liu

Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection (2025.coling-main)

Challenge:	Existing top-performing methods for Lexical Relation Mining rely on pre-trained language models yet fail to distinguish nuanced lexical relations.
Approach:	They propose a framework to leverage structured sememe knowledge to enhance LRC and LE.
Outcome:	The proposed method outperforms existing methods on benchmarks and outperformed the LLMs.

Copied to clipboard

Dmitri Roussinov, Serge Sharoff, Nadezhda Puchnina

Information Retrieval

Challenge:	Recent advances in Large Language Models (LLMs) have pushed the boundaries of natural language processing, but their consistency is often limited when applied to unfamiliar domains.
Approach:	They propose a method that controls which predictive indicators are used and which are excluded during classification.
Outcome:	The proposed method reduces the OOD gap by up to 20 percentage points in a few-shot setup.

Data-Centric Explainable Debiasing for Improving Fairness in Pre-trained Language Models (2024.findings-acl)

Copied to clipboard

Yingji Li, Mengnan Du, Rui Song, Xin Wang, Ying Wang

UMSE: Unified Multi-scenario Summarization Evaluation (2023.findings-acl)

Challenge:	Existing data-centric debiasing strategies mainly leverage explicit bias words for counterfactual data augmentation to balance the training data.
Approach:	They propose a method which uses an explainability method to search for implicit bias words to assist in debiasing PLMs.
Outcome:	Extensive results show that the proposed method achieves state-of-the-art debiasing performance and strong generalization while maintaining predictive abilities.

Copied to clipboard

Shen Gao, Zhitao Yao, Chongyang Tao, Xiuying Chen, Pengjie Ren, Zhaochun Ren, Zhumin Chen

RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024.findings-naacl)

Challenge:	Summarization quality evaluation is a non-trivial task in text summarization.
Approach:	They propose a unified multi-scenario summarization evaluation model that shares cross-sceenario knowledge and uses a self-supervised training paradigm to optimize the model without extra human labeling.
Outcome:	The proposed model can achieve comparable performance with existing methods for three evaluation scenarios.

Copied to clipboard

Javad Rafiei Asl, Prajwal Panzade, Eduardo Blanco, Daniel Takabi, Zhipeng Cai

PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models (2022.emnlp-main)

Challenge:	Existing pre-trained language models exhibit poor generalization and robustness in adversarial settings.
Approach:	They propose a self-supervised sentence embedding framework that improves generalization and robustness against adversarial attacks.
Outcome:	The proposed framework reduces the success rate of adversarial attacks by almost half . it also improves semantic text similarity tasks and various transfer tasks .

Copied to clipboard

Yupeng Zhang, Hongzhi Zhang, Sirui Wang, Wei Wu, Zhoujun Li

Path-enhanced Pre-trained Language Model for Knowledge Graph Completion (2025.findings-emnlp)

Challenge:	A wide range of NLP tasks benefit from fine-tuning of pretrained language models (PLMs), however, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine- tuned model.
Approach:	They propose a noisy training mechanism which considers each parameter’s importance in the downstream task to help fine-tune pretrained language models.
Outcome:	The proposed method can be used to fine-tune pretrained language models on a wide range of tasks and consistently achieve higher performance.

Copied to clipboard

Hao Wang, Dandan Song, Zhijing Wu, Yuhang Tian, Pan Yang

TextMixer: Mixing Multiple Inputs for Privacy-Preserving Inference (2023.findings-emnlp)

Challenge:	Pre-trained language models have achieved remarkable knowledge graph completion (KGC) success.
Approach:	They propose a path-enhanced pre-trained language model-based knowledge graph completion method which uses multi-view generation to infer missing facts in triple-level and path-level simultaneously.
Outcome:	The proposed method significantly improves the performance of the knowledge graph completion task.

Copied to clipboard

Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang

BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation (2022.emnlp-main)

Challenge:	Pre-trained language models (PLMs) are often deployed as cloud services, enabling users to upload textual data and perform inference remotely.
Approach:	They propose a privacy-preserving inference framework called MixPi which aims to obfuscate a user's private input by mixing it with multiple other inputs.
Outcome:	The proposed framework surpasses existing privacy-preserving methods on token and sentence classification tasks.

Copied to clipboard

Tianxiang Sun, Junliang He, Xipeng Qiu, Xuanjing Huang

HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification (2022.emnlp-main)

Challenge:	Currently, pre-trained language model (PLM) based metrics are widely adopted in text generation tasks.
Approach:	They propose to use PLMs to encode stereotypical societal biases in PLM-based metrics . they show that popular metrics exhibit higher social bias than traditional metrics based on 6 attributes .
Outcome:	The proposed method shows that PLM-based metrics exhibit higher social bias than traditional metrics on 6 attributes.

Copied to clipboard

Zihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui, Houfeng Wang

HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression (2021.emnlp-main)

Challenge:	Hierarchical text classification (HTC) is a multi-label classification problem with a complex label hierarchy.
Approach:	They propose a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label perspective using a dynamic virtual template and label words that take the form of soft prompts to fuse the label hierarchy knowledge.
Outcome:	The proposed method achieves state-of-the-art performance on 3 popular HTC datasets and is adept at handling imbalance and low resource situations.

Copied to clipboard

Chenhe Dong, Yaliang Li, Ying Shen, Minghui Qiu

Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021.acl-long)

Challenge:	Large pre-trained language models (PLMs) have shown overwhelming performances on many tasks, but their large size and slow inference speed have hindered practical deployments.
Approach:	They propose a hierarchical relational knowledge distillation method to capture hierarchic and domain relational information.
Outcome:	The proposed method outperforms existing methods on multi-domain datasets and is highly reproducible.

Copied to clipboard

Cunxiang Wang, Pai Liu, Yue Zhang

SETI: Systematicity Evaluation of Textual Inference (2023.findings-acl)

Challenge:	Existing work is limited in using small benchmarks with high test-train overlaps.
Approach:	They construct a dataset of closed-book QA using SQuAD and investigate the performance of BART.
Outcome:	Experiments show that pre-trained language models can achieve high performance on closed-book QA tasks.

Copied to clipboard

Xiyan Fu, Anette Frank

Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)

Challenge:	Existing pre-trained language models (PLMs) have shown remarkable performance on this task, but little is known about their ability to address compositional generalization.
Approach:	They propose a benchmark to evaluate pre-trained language models' systematicity in the domain of textual inference.
Outcome:	The proposed benchmark evaluates pre-trained language models on six widely used PLMs.

Copied to clipboard

Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani

FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking (2025.findings-naacl)

Challenge:	Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score.
Approach:	They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers.
Outcome:	The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks.

Copied to clipboard

Jabez Magomere, Elena Kochkina, Samuel Mensah, Simerjot Kaur, Charese Smiley

Enhancing Language Representation with Constructional Information for Natural Language Understanding (2023.acl-long)

Challenge:	FinNLI is a benchmark dataset for Financial Natural Language Inference (NLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts.
Approach:	They propose to use FinNLI to evaluate financial natural language inference models across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts.
Outcome:	The proposed dataset is based on a high-quality test set of 3,304 instances annotated by finance experts.

Copied to clipboard

Lvxiaowei Xu, Jianwang Wu, Jiawei Peng, Zhilin Gong, Ming Cai, Tianxiang Wang

Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations (2023.acl-long)

Challenge:	Recent advances in natural language processing focus on acquiring lexico-semantic information.
Approach:	They propose a construction grammar which highlights the pairings of form and meaning to enrich language representation.
Outcome:	The proposed model is superior to existing models on a variety of NLU tasks.

Copied to clipboard

Linlin Liu, Xingxuan Li, Megh Thakkar, Xin Li, Shafiq Joty, Luo Si, Lidong Bing

Semantics or spelling? Probing contextual word embeddings with orthographic noise (2024.findings-acl)

Challenge:	Using hidden representations, pretrained language models are prone to overfitting due to the huge amount of parameters.
Approach:	They propose a method that inserts random autoencoders between hidden layers of a PLM to transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers.
Outcome:	The proposed method improves performance across sequence- and token-level lowresource tasks.

Copied to clipboard

Jacob Matthews, John Starr, Marten Schijndel

Neural Pipeline for Zero-Shot Data-to-Text Generation (2022.acl-long)

Challenge:	Pretrained language models (PLMs) are used to generate contextual word embeddings . linguistics research has focused on semantic information in hidden states .
Approach:	They investigate whether a single character swap in the input word will not affect the resulting representation . they find that PLM-derived contextual word embeddings are highly sensitive to noise .
Outcome:	The results show that the PLM-derived representations are highly sensitive to noise . the fewer tokens used to represent a word at input, the more sensitive their CWE is .

Copied to clipboard

Zdeněk Kasner, Ondrej Dusek

Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)

Challenge:	In data-to-text generation, training on in-domain data leads to overfitting and repeating training data noise.
Approach:	They propose to train pretrained language models on general-domain text-based operations by transforming single-item descriptions with modules trained on ordering, aggregation, and paragraph compression.
Outcome:	The proposed approach enables D2T generation from RDF triples in zero-shot settings.

Copied to clipboard

Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang

CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)

Challenge:	Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks.
Approach:	They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem.
Outcome:	The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings.

Copied to clipboard

Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu

Addressing Healthcare-related Racial and LGBTQ+ Biases in Pretrained Language Models (2024.findings-naacl)

Challenge:	Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding.
Approach:	They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning .
Outcome:	The proposed model can encode words into fine-grained representations without modification of production pipelines.

Copied to clipboard

Sean Xie, Saeed Hassanpour, Soroush Vosoughi

Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Models (2023.findings-acl)

Challenge:	Pretrained language models (PLMs) propagate social stigmas and stereotypes, a critical concern given their widespread use.
Approach:	They adapt two intrinsic bias benchmarks to quantify racial and LGBTQ+ biases in prevalent PLMs and empirically evaluate the effectiveness of various debiasing methods in mitigating these biase.
Outcome:	The proposed methods reduce biases without compromising performance in downstream tasks.

Copied to clipboard

Fengzhu Zeng, Wei Gao

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words (2021.acl-long)

Challenge:	Existing methods for few-shot and zero-shot fact verification require a large set of training data.
Approach:	They propose a method to prompt pre-trained language models to be consistent to improve the factuality assessment capability of PLMs.
Outcome:	The proposed method outperforms state-of-the-art few-shot fact verification models with a small number of unlabeled instances on zero-shot verification.

Copied to clipboard

Valentin Hofmann, Janet Pierrehumbert, Hinrich Schütze

Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training (2024.lrec-main)

Challenge:	Pretrained language models (PLMs) are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as bytepair encoding.
Approach:	They propose to use BERT as an example PLM to study its semantic representations of English derivatives to test their hypothesis.
Outcome:	The proposed model outperforms BERT on a series of semantic probing tasks.

Copied to clipboard

Longhui Zhang, Dingkun Long, Meishan Zhang, Yanzhao Zhang, Pengjun Xie, Min Zhang

Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)

Challenge:	Pretrained language models (PLMs) have been successful in addressing word boundaries in Chinese sequence labeling tasks, but they rarely consider boundary information explicitly.
Approach:	They propose a method to integrate unsupervised boundary information into Chinese BERT's pre-training objectives and a supervised boundary-aware PLM.
Outcome:	The proposed model outperforms the vanilla version on Chinese sequence labeling tasks and in broader Chinese natural language understanding tasks.

Copied to clipboard

Jiaxi Li, Wei Lu

Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models (2023.findings-emnlp)

Challenge:	a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers .
Approach:	They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span .
Outcome:	The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings.

Copied to clipboard

Amirhossein Kazemnejad, Mehdi Rezagholizadeh, Prasanna Parthasarathi, Sarath Chandar

Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)

Challenge:	Recent research has demonstrated that pre-trained language models acquire a broad range of knowledge about linguistic structures, encyclopedic relations, levels of commonsense, and even coding and reasoning rules.
Approach:	They propose a systematic framework to measure parametric knowledge utilization in pre-trained language models by extracting parametric information from a PLM and constructing a downstream task around this extracted knowledge.
Outcome:	The proposed framework extracts parametric knowledge from a PLM and constructs a downstream task around this extracted knowledge.

Copied to clipboard

Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao

SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation (2024.findings-naacl)

Challenge:	Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive .
Approach:	They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training .
Outcome:	The proposed framework achieves superior performance compared with baselines.

Copied to clipboard

Shasha Guo, Lizi Liao, Jing Zhang, Yanling Wang, Cuiping Li, Hong Chen

Knowledge Inheritance for Pre-trained Language Models (2022.naacl-main)

Challenge:	Existing methods have significantly boosted the performance of Knowledge Base Question Generation (KBQG) through pre-trained language models thanks to the richly endowed semantic knowledge.
Approach:	They propose a framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG by combining skeleton heuristic guidance with a soft prompting approach.
Outcome:	The proposed framework incorporates "skeleton heuristics" which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions.

Copied to clipboard

Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

MUX-PLMs: Data Multiplexing for High-throughput Language Models (2023.findings-emnlp)

Challenge:	Existing large-scale pre-trained language models are mainly trained from scratch individually, ignoring that many well-taught PLMs are available.
Approach:	They propose a pre-training framework called knowledge inheritance and propose auxiliary supervision to efficiently learn larger PLMs.
Outcome:	The proposed framework can be used to train large-scale language models with huge parameters and a large dataset can be adapted to domain adaptation and knowledge transfer.

Copied to clipboard

Vishvak Murahari, Ameet Deshpande, Carlos Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, Karthik Narasimhan

KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters (2024.findings-acl)

Challenge:	MUX-PLMs are high-throughput pre-trained language models that can be fine-tuned for any downstream task to yield high-performance.
Approach:	They propose to train language models with data multiplexing to achieve 2x/5x inference speedup . they use multiplexers to entangle and disentangle inputs to achieve the same performance .
Outcome:	MUX-PLMs achieve 2x/5x inference speedup with 1-4 % drop on broad suite of tasks.

Copied to clipboard

SungHo Kim, Juhyeong Park, Yeachan Kim, SangKeun Lee

Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation (2023.findings-emnlp)

Challenge:	Existing pre-trained language models for Korean have overlooked character representation principles . a new framework for Korean PLMs uses the invention principles of Hangeul to represent letters .
Approach:	They propose a framework that uses Hangeul's invention principles to represent character in Korean . their method outperforms the state-of-the-art Korean PLM by 2.11% in five tasks .
Outcome:	The proposed method outperforms the state-of-the-art Korean PLM by 2.11% in five Korean natural language understanding tasks.

Copied to clipboard

Tianyu Yang, Thy Tran, Iryna Gurevych

Dialogue Systems & Conversational Agents

Challenge:	Existing variational dialog models have pre-trained, restricting diversity of responses . a diffusion model increases complexity of prior distribution and its compatibility with PLMs .
Approach:	They propose a hierarchical conditional variational autoencoder with diffusion priors to address these challenges.
Outcome:	The proposed method generates more diverse responses without dialog pre-training.

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)

Copied to clipboard

Kun Zhou, Wayne Xin Zhao, Sirui Wang, Fuzheng Zhang, Wei Wu, Ji-Rong Wen

Probing for Predicate Argument Structures in Pretrained Language Models (2022.acl-long)

Challenge:	Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks.
Approach:	They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity.
Outcome:	The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks.

Copied to clipboard

Simone Conia, Roberto Navigli

Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022.emnlp-main)

Challenge:	Recent proposed approaches have achieved impressive results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL)
Approach:	They propose to probe for predicate argument structures in pretrained language models . they show that PLMs encode semantic structures directly into contextualized representations .
Outcome:	The proposed models have achieved impressive results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL)

Copied to clipboard

Chenghao Yang, Xuezhe Ma

ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024.acl-long)

Challenge:	Recent studies have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs.
Approach:	They propose a component-wise gradient norm clipping method to adjust convergence speed for different components to improve generalization performance, convergence speed, and training stability.
Outcome:	The proposed method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability.

Copied to clipboard

Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua

BECEL: Benchmark for Consistency Evaluation of Language Models (2022.coling-1)

Challenge:	Language Models excel in understanding textual descriptions of proteins, but struggle to process texts.
Approach:	They propose a framework for Protein-to-Text Generation for Text-based Protein Understanding that integrates a PLM as its protein understanding module.
Outcome:	The proposed framework surpasses existing baselines and is highly efficient in protein-to-text generation.

Copied to clipboard

Myeongjun Jang, Deuk Sin Kwon, Thomas Lukasiewicz

Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models (2021.findings-emnlp)

Challenge:	Existing definitions of behavioural consistency are inconsistent across many studies.
Approach:	They propose a behavioural consistency model and propose behavioural taxonomy that classifies consistencies into several sub-categories.
Outcome:	The proposed model performs poorly on 19 test cases while exhibiting high inconsistency in many cases.

Copied to clipboard

Qiuhao Lu, Dejing Dou, Thien Huu Nguyen

CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation (2023.findings-acl)

Challenge:	Existing domain-specific pre-trained language models (PLMs) rely on self-supervised learning over large amounts of domain text, without explicitly integrating domain- specific knowledge.
Approach:	They propose to integrate domain knowledge from diverse sources into PLMs by using adapters that are pre-trained for individual domain knowledge sources and integrated via an attention-based knowledge controller.
Outcome:	The proposed architecture integrates domain knowledge from diverse sources into PLMs in a parameter-efficient way.

Copied to clipboard

YunSeok Choi, Jee-Hyong Lee

Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter (2023.findings-acl)

Challenge:	Prompt-tuning methods have been used to solve inefficient parameter update and storage issues in Natural Language Generation tasks.
Approach:	They propose a task-agnostic prompt tuning method that reflects the traits of PLM for program language.
Outcome:	The proposed method is effective in three PLG tasks, not only in the full-data setting but also in the low-resource setting and cross-domain setting.

Copied to clipboard

Yi Liu, Xiaohan Bi, Lei Li, Sishuo Chen, Wenkai Yang, Xu Sun

Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models (2022.acl-long)

Challenge:	Existing frameworks for federated multilingual neural machine translation (Fed-MNMT) are limited in language resources.
Approach:	They propose a framework that keeps PLMs frozen and only transfers lightweight adapter modules between clients.
Outcome:	The proposed framework reduces communication cost by over 98% while achieving similar or even better performance compared to baselines.

Copied to clipboard

Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, Nigel Collier

Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)

Challenge:	Despite the growing progress of probing knowledge for pre-trained language models, specialised areas such as the biomedical domain are vastly under-explored.
Approach:	They propose a biomedical knowledge probing benchmark, MedLAMA, constructed based on the Unified Medical Language System (UMLS) Metathesaurus.
Outcome:	The proposed approach pushes the acc@10 to 28%, but the performance gap remains notable.

Copied to clipboard

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong

Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis (2022.emnlp-main)

Challenge:	Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory.
Approach:	They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules.
Outcome:	The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin.

Copied to clipboard

Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, JIan Guo, Nan Duan

Sentiment Analysis

Challenge:	Existing pre-trained language representation models (PLMs) capture sentiment information from word-level while under-considering sentence-level information.
Approach:	They propose a Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks that enhance the PLM’s knowledge about sentiment words.
Outcome:	The proposed model achieves state-of-the-art on various sentence-level and aspect-level sentiment classification benchmarks.

COPEN: Probing Conceptual Knowledge in Pre-trained Language Models (2022.emnlp-main)

Copied to clipboard

Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, Qun Liu

PEMA: An Offsite-Tunable Plug-in External Memory Adaptation for Language Models (2024.naacl-long)

Challenge:	Existing knowledge probing studies focus on evaluating factual knowledge of pre-trained language models (PLMs) but ignore conceptual knowledge.
Approach:	They evaluate conceptual knowledge of pre-trained language models by annotating 24k data instances covering 393 concepts.
Outcome:	The proposed tasks evaluate pre-trained language models' conceptual knowledge of entities, learn conceptual properties, and conceptualize entities in contexts.

Copied to clipboard

HyunJin Kim, Young Jin Kim, JinYeong Bak

DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction (2023.findings-acl)

Challenge:	Pre-trained language models (PLMs) show impressive performance in various downstream NLP tasks.
Approach:	They propose a Parameter-Efficient Fine-Tuning method that integrates with context representations from test data to perform downstream tasks.
Outcome:	The proposed method outperforms other methods in memory and latency efficiency and maintains sentence meaning and generating appropriate language and styles.

Copied to clipboard

Bo Lv, Xin Liu, Shaojie Dai, Nayu Liu, Fan Yang, Ping Luo, Yue Yu

MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension (2023.findings-emnlp)

Challenge:	Prompt-based methods have shown their efficacy in transferring general knowledge within pre-trained language models (PLMs) however, when applied to zero-shot entity and relation extraction, they struggle with the limited coverage of verbalizers to labels and the slow inference speed.
Approach:	They propose a method which reformulates zero-shot tasks into token discrimination tasks without having to construct verbalizers.
Outcome:	The proposed method outperforms baselines on two zero-shot entity recognition datasets with higher inference speed and achieves 7.5% improvement over previous state-of-the-art models on Wiki-ZSL and FewRel.

Copied to clipboard

Guoxin Chen, Yiming Qian, Bowen Wang, Liangzhi Li

Large Product Key Memory for Pretrained Language Models (2020.findings-emnlp)

Challenge:	Existing soft prompt methods focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset.
Approach:	They propose a multi-level prompt tuning method that utilizes prompts at task-specific, domain-specific and context-specific levels to enhance the comprehension of input semantics.
Outcome:	The proposed method improves on 12 benchmarks on various QA formats and achieves an average improvement of 1.94% over the state-of-the-art methods.

Copied to clipboard

Gyuwan Kim, Tae Hwan Jung

Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)

Challenge:	Existing product key memory (PKM) models that increase model capacity with insignificant computational overhead are limited to causal language modeling.
Approach:	They propose product key memory (PKM) that enables very efficient and exact nearest neighbor search in a large number of learnable memory slots.
Outcome:	The proposed product key memory improves model capacity and performance by replacing a feed-forward network with a model weighted model.

Copied to clipboard

Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, Ali Ghodsi, Phillippe Langlais

Fast and Accurate End-to-End Span-based Semantic Role Labeling as Word-based Graph Parsing (2022.coling-1)

Challenge:	Recent studies show that the flatness of the local minimum correlates well with better generalization.
Approach:	They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs.
Outcome:	The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost.

Copied to clipboard

Shilin Zhou, Qingrong Xia, Zhenghua Li, Yu Zhang, Yu Hong, Min Zhang

Prompting Contrastive Explanations for Commonsense Reasoning Tasks (2021.findings-acl)

Challenge:	Using end-to-end span-based SRL, we propose a word-based graph parsing task for word-level representation of spans . compared with word-driven SRL, span-Based SRL is more complex due to difficulties in determining argument boundaries.
Approach:	They propose to cast end-to-end span-based SRL as a word-based graph parsing task . they propose a constrained Viterbi procedure to ensure the legality of the output graph .
Outcome:	The proposed model can parse 669/252 sentences per second without and with pre-trained models.

Copied to clipboard

Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, Luke Zettlemoyer

Prompting Language Models for Linguistic Structure (2023.acl-long)

Challenge:	Large pretrained language models (PLMs) can achieve near-human performance on commonsense reasoning tasks, but provide little human-interpretable evidence of the underlying reasoning they use.
Approach:	They propose to use large pretrained language models to generate evidence for commonsense reasoning NLP tasks . they use models to contrast alternative explanations based on key attribute(s) required to justify the correct answer .
Outcome:	The proposed model improves performance on two commonsense reasoning benchmarks compared to previous non-contrastive alternatives.

Copied to clipboard

Terra Blevins, Hila Gonen, Luke Zettlemoyer

Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation (2022.emnlp-main)

Challenge:	Existing prompting methods can test this hypothesis on autoregressive PLMs.
Approach:	They propose a structured prompting approach for linguistic structured prediction tasks that performs zero- and few-shot sequence tagging with autoregressive PLMs.
Outcome:	The proposed approach shows that the model can perform few-shot sequence tagging on part-of-speech taging, named entity recognition, and sentence chunking tasks.

Copied to clipboard

Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, Qi Liu

ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications (2024.naacl-long)

Challenge:	Existing work fine tunes the PLM with the news recommendation task, which can cause a domain shift problem.
Approach:	They propose a self-supervised method to adapt general PLM to news domain with a contrastive matching task between news titles and news bodies.
Outcome:	The proposed method can improve both the effectiveness and efficiency of the large PLM-based news recommendation model while maintaining its performance.

Copied to clipboard

Sotaro Takeshita, Tommaso Green, Ines Reinig, Kai Eckert, Simone Ponzetto

Summarization

Challenge:	Existing statistical phrasal or hierarchical machine translation systems relies on a large set of translation rules which results in engineering challenges.
Approach:	They propose to use factorized grammar from the field of linguistics as more general translation rules from XTAG English Grammar to generate a manually crafted summarization dataset.
Outcome:	The proposed method outperforms existing methods on low-resource language translation tasks with less training data.

Prix-LM: Pretraining for Multilingual Knowledge Base Construction (2022.acl-long)

Copied to clipboard

Wenxuan Zhou, Fangyu Liu, Ivan Vulić, Nigel Collier, Muhao Chen

DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs (2024.findings-emnlp)

Challenge:	Existing methods to build and enrich multilingual knowledge bases have not been successful . knowledge expressed in different languages may be complementary and unequally distributed .
Approach:	They propose a model that integrates useful multilingual and KB-based factual knowledge into a single model.
Outcome:	The proposed model can provide richer combined knowledge than monolingual KBs.

Copied to clipboard

Divya Jyoti Bajpai, Manjesh Hanawal

From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment (2026.findings-acl)

Challenge:	Pre-trained Language Models (PLMs) exhibit good accuracy and generalization ability but their large size results in high inference latency.
Approach:	They propose an unsupervised domain adaptation framework that employs knowledge distillation to achieve domain-invariant representations at each layer.
Outcome:	The proposed framework outperforms early exit methods and domain adaptation methods under domain shift scenarios.

Copied to clipboard

Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci

KALA: Knowledge-Augmented Language Model Adaptation (2022.naacl-main)

Challenge:	a framework for sentence-level interpretability of rubric-based scoring is proposed . aaron e. smith: automated scoring models provide little insight into why scores are produced .
Approach:	They propose a framework for sentence-level interpretability of rubric-based scoring that combines Shapley-value attributions with rationales generated by large language models.
Outcome:	The proposed framework compares fine-tuned pretrained language models with large language models . it shows that fine- tuned models outperform LLMs in prediction accuracy but exhibit label compression toward mid-scale scores .

Copied to clipboard

Minki Kang, Jinheon Baek, Sung Ju Hwang

Tomoyuki Jinno, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Challenge:	Pre-trained language models (PLMs) have proved to be effective on various natural language understanding tasks.
Approach:	They propose a domain adaption framework which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts.
Outcome:	The proposed framework outperforms adaptive pre-training on question answering and named entity recognition tasks on multiple datasets across different domains.

Copied to clipboard

Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning (2022.coling-1)

Challenge:	Existing knowledge probes for pre-trained language models exhibit quadratic time complexity, limiting the size of knowledge graphs used for probing.
Approach:	They propose an embedding-based relational probe that evaluates pre-trained language models' factual knowledge retrieval capabilities.
Outcome:	The proposed probe achieves effective time complexity of linear order O(n), supports rank-based evaluation metrics including Hit@k, handles multi-token entity names and enables probing whilst disambiguating homographic tail-entity names.

Copied to clipboard

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, Dietrich Klakow

On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)

Challenge:	Multilingual pre-trained language models have shown impressive performance on several downstream tasks for both high-resourced and low-resource languages.
Approach:	They propose to apply multilingual adaptive fine-tuning to 17 most-resourced African languages and three other high-resource languages to encourage cross-lingual transfer learning.
Outcome:	The proposed approach is competitive to LAFT on individual languages while requiring significantly less disk space.

Copied to clipboard

Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, Yutaka Matsuo

KLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships (2021.findings-emnlp)

Challenge:	Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism.
Approach:	They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models.
Outcome:	The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages.

Copied to clipboard

Lei He, Suncong Zheng, Tao Yang, Feng Zhang

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers (2021.findings-acl)

Challenge:	Existing knowledge-enhanced pretrained language models focus on entity information and ignore fine-grained relationships between entities.
Approach:	They propose to incorporate KG into the language learning process to obtain a KG-enhanced pretrained Language Model.
Outcome:	The proposed model improves on several knowledge-driven tasks, such as entity typing and relation classification, compared with the state-of-the-art knowledge-enhanced PLMs.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Yongfeng Huang

AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising (2025.naacl-long)

Challenge:	Pre-trained language models (PLMs) have huge model sizes and computational complexity, making it difficult to deploy them to low-latency and high-concurrence online systems.
Approach:	They propose a multi-teacher knowledge distillation framework for pre-trained language model compression that can train high-quality student model from multiple teacher PLMs.
Outcome:	The proposed framework can train high-quality student model from multiple teacher PLMs with shared pooling and prediction layers to align output space for better collaborative teaching.

Copied to clipboard

Peinan Zhang, Yusuke Sakai, Masato Mita, Hiroki Ouchi, Taro Watanabe

From Text Segmentation to Enhanced Representation Learning: A Novel Approach to Multi-Label Classification for Long Texts (2024.findings-emnlp)

Challenge:	Existing pre-trained language models outperform them in certain domains, indicating that there is significant potential for further improvement in this area.
Approach:	They propose to use pre-trained language models to evaluate ad texts from multiple perspectives within real-world advertising operations to define five tasks and construct a Japanese dataset.
Outcome:	The proposed benchmark outperforms existing pre-trained language models in several tasks, but humans outperformed them in certain domains.

Copied to clipboard

Wang Zhang, Xin Wang, Qian Wang, Tao Deng, Xiaoru Wu

Probing Simile Knowledge from Pre-trained Language Models (2022.acl-long)

Challenge:	Existing models rely on pre-trained language models, which have a maximum input sequence length of 512 tokens, and therefore have 'input length limitation'.
Approach:	They propose a text segmentation algorithm which guarantees to produce the optimal segmentation to address the issue of input length limitation caused by PLMs.
Outcome:	The proposed method improves both text and label representations on MLTC datasets, unraveling the intricate correlations between texts and labels.

Copied to clipboard

Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, Chang Su

Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)

Challenge:	Existing approaches to learn generic knowledge from a large corpus are time-consuming and labor-intensive.
Approach:	They propose a framework to probe simile knowledge from pre-trained language models to solve SI and SG tasks.
Outcome:	The proposed framework solves the SI and SG tasks in a simile triple completion task.

Copied to clipboard

Dongsuk Oh, Yejin Kim, Hodong Lee, H. Howie Huang, Heuiseok Lim

PclGPT: A Large Language Model for Patronizing and Condescending Language Detection (2024.findings-emnlp)

Challenge:	Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks.
Approach:	They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks.
Outcome:	The proposed method improves on standard semantic textual similarity and semantic search tasks.

Copied to clipboard

Hongbo Wang, LiMingDa LiMingDa, Junyu Lu, Hebin Xia, Liang Yang, Bo Xu, Ruizhu Liu, Hongfei Lin

Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023.emnlp-main)

Challenge:	Patronizing and condescending language is an essential branch of toxic language . pre-trained language models perform poorly in detecting PCL due to its implicit toxicity traits .
Approach:	They propose a novel LLM benchmark for patronizing and condescending language . they use a dataset to analyze the toxicity of patronizing condescending languages .
Outcome:	The proposed model can detect patronizing and condescending language (PCL) the model can be used to analyze the toxicity of the language and to improve the detection.

Copied to clipboard

Di Wu, Wasi Ahmad, Kai-Wei Chang

Sustainable Modular Debiasing of Language Models (2021.findings-emnlp)

Challenge:	Keyphrase generation is a longstanding task in NLP with widespread applications.
Approach:	They propose a likelihood-based decode-select algorithm for seq2seq PLMs that improves greedy search by an average of 4.7% semantic F1 across five datasets.
Outcome:	The proposed algorithm improves greedy search by an average of 4.7% semantic F1 across five datasets.

Copied to clipboard

Anne Lauscher, Tobias Lueken, Goran Glavaš

Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge (2022.acl-long)

Challenge:	Existing debiasing methods modify all of the PLM parameters, which is costly and leads to (catastrophic) forgetting of useful language knowledge.
Approach:	They propose a modular debiasing approach based on dedicated adapters that inject adapter modules into the original PLM layers and update only the adapters.
Outcome:	The proposed approach is based on dedicated adapters and retains fairness even after large-scale training.

Copied to clipboard

Linhai Zhang, Xuemeng Hu, Boyu Wang, Deyu Zhou, Qian-Wen Zhang, Yunbo Cao

Accelerating Inference for Pretrained Language Models by Unified Multi-Perspective Early Exiting (2022.coling-1)

Challenge:	Recent studies have shown that using external knowledge such as pre-trained word embeddings or pre-train language models only achieved limited performance improvements but with huge computational overhead.
Approach:	They propose to incorporate external knowledge into neural topic modeling by pre-trained word embeddings (PWEs) or pre-train language models (PLMs) they propose to fine-tune the neural topic model on the target dataset and reduce the huge size of training data.
Outcome:	The proposed approach outperforms current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs on three datasets and greatly reduces the huge size of training data.

Copied to clipboard

Jun Kong, Jin Wang, Liang-Chih Yu, Xuejie Zhang

Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)

Challenge:	Existing competitive methods to accelerate inference of pretrained language models are limited by their complexity and computational consumption.
Approach:	They propose a unified horizontal and vertical multi-perspective early exiting framework to accelerate inference of transformer-based models.
Outcome:	Experiments show that MPEE can achieve higher acceleration inference with competent performance than existing competitive methods.

Copied to clipboard

Zhaochen Su, Juntao Li, Zikang Zhang, Zihan Zhou, Min Zhang

Order-Agnostic Data Augmentation for Few-Shot Named Entity Recognition (2024.acl-long)

Challenge:	Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components.
Approach:	They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens.
Outcome:	The proposed method outperforms existing temporal language models on two datasets and three tasks.

Copied to clipboard

Huiming Wang, Liying Cheng, Wenxuan Zhang, De Wen Soh, Lidong Bing

Detecting Cybercrimes in Accordance with Pakistani Law: Dataset and Evaluation Using PLMs (2024.lrec-main)

Challenge:	Existing DA methods for named entity recognition (NER) are costly and labor-intensive to acquire, necessitating innovative approaches to data scarcity.
Approach:	They propose an order-agnostic data augmentation solution that exploits the order-based property in the training phase of sequence-to-sequence NER methods for data augmented.
Outcome:	The proposed method significantly enhances the few-shot capabilities of pre-trained language models in low-resource settings.

Copied to clipboard

Faizad Ullah, Ali Faheem, Ubaid Azam, Muhammad Sohaib Ayub, Faisal Kamiran, Asim Karim

CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model (2022.findings-emnlp)

Challenge:	Roman Urdu is a widely used language in Pakistan but lacks sufficient resources and tools for text-based cybercrime detection.
Approach:	They propose to use a benchmark dataset for text-based cybercrime detection in Roman Urdu to improve the performance of pre-trained language models.
Outcome:	The proposed model achieves the highest performance on all metrics.

Copied to clipboard

Shang-Hsuan Chiang, Ssu-Cheng Wang, Yao-Chung Fan

John praised Mary because _he_? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs (2021.findings-acl)

Challenge:	Existing approaches to generate cloze distractors with carefully-designed distractors are limited due to wrong option selection.
Approach:	They propose to employ pre-trained language models as an alternative to cloze distractor generation by using pre-designed distractors.
Outcome:	The proposed model improves the state-of-the-art cloze test score from 14.94 to 34.17 (NDCG@10) The proposed framework improves clozing distractors by incorporating pre-trained language models.

Copied to clipboard

Yova Kementchedjhieva, Mark Anderson, Anders Søgaard

G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks (2022.emnlp-main)

Challenge:	Psycholinguists have identified one such cue in the implicit causality bias of interpersonal verbs.
Approach:	They propose to use pre-trained language models to encode IC bias at inference time . they hypothesize that when a cause is explicitly stated, an incongruent IC biased leads to a delay in human processing.
Outcome:	The results suggest that pre-trained language models tend to prioritize lexical patterns over higher-order signals.

Copied to clipboard

Zhongwei Wan, Yichun Yin, Wei Zhang, Jiaxin Shi, Lifeng Shang, Guangyong Chen, Xin Jiang, Qun Liu

DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)

Challenge:	Existing domain-adaptive pre-training (DAPT) models tend to forget the general knowledge acquired by general PLMs, leading to catastrophic forgetting and sub-optimal performance.
Approach:	They propose a framework which augments the domain-specific PLM by a memory built from the frozen general PLM without losing the general knowledge.
Outcome:	The proposed framework augments the domain-specific PLM by a memory built from the frozen general PLM without losing the general knowledge.

Copied to clipboard

Wenxuan Zhang, Min Huang, Zhuoyang Song, Qinghai Miao

CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding (2023.emnlp-main)

Challenge:	Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive.
Approach:	They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality.
Outcome:	The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters.

Copied to clipboard

Yixiao Ma, Yueyue Wu, Weihang Su, Qingyao Ai, Yiqun Liu

Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation (2022.findings-emnlp)

Challenge:	Existing legal-oriented PLMs rely on replacing general domain training data with legal data or extending the input length to fit the long-length characteristic of legal data.
Approach:	They propose a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases.
Outcome:	The proposed model outperforms existing general domain pre-training models and legal-specific pre-trainers on multiple benchmarks.

Copied to clipboard

Xueliang Zhao, Yuxuan Wang, Chongyang Tao, Chenshuo Wang, Dongyan Zhao

Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)

Challenge:	Existing methods for video-grounded dialogue generation do not allow information from different modalities to complement each other.
Approach:	They propose a video-grounded dialogue generation model that integrates video data into pre-trained language models to allow information from different modalities to complement each other.
Outcome:	The proposed model outperforms state-of-the-art models on automatic and human evaluations on two public datasets.

Copied to clipboard

Sheng Liang, Philipp Dufter, Hinrich Schütze

MaCSC: Towards Multimodal-augmented Pre-trained Language Models via Conceptual Prototypes and Self-balancing Calibration (2024.naacl-long)

Challenge:	Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace.
Approach:	They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT.
Outcome:	The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better.

Copied to clipboard

Xianwei Zhuang, Zhichang Wang, Xuxin Cheng, Yuxin Xie, Liming Liang, Yuexian Zou

Distill, Fuse, Pre-train: Towards Effective Event Causality Identification with Commonsense-Aware Pre-trained Model (2024.lrec-main)

Challenge:	Existing approaches to training pre-trained language models (PLMs) focus on static image modality; inevitably encounter modality gaps and noise; and treat all modalities.
Approach:	They propose a multimodal-augmented framework that can infuse multimodal semantics into PLMs and facilitate a self-balancing calibration of information allocation.
Outcome:	The proposed framework outperforms baselines on multiple NLP tasks and outperformed existing frameworks.

Copied to clipboard

Peixin Huang, Xiang Zhao, Minghao Hu, Zhen Tan, Weidong Xiao

Exploring Mode Connectivity for Pre-trained Language Models (2022.emnlp-main)

Challenge:	Existing methods to detect causal relationships in unstructured texts ignore trivial knowledge which may prejudice performance.
Approach:	They propose a pipeline to build a commonsense-aware pre-trained model which integrates reliable task-specific knowledge from commonsens graphs.
Outcome:	The proposed pipeline integrates reliable task-specific knowledge from commonsense graphs.

Copied to clipboard

Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, Jie Zhou

From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning (2023.emnlp-main)

Challenge:	Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found.
Approach:	They investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path.
Outcome:	The proposed model can be used to find low-loss paths between two minima, and to understand how their mode connectivity affects their task knowledge.

Copied to clipboard

Zheyuan Zhang, Shane Storks, Fengyuan Hu, Sungryull Sohn, Moontae Lee, Honglak Lee, Joyce Chai

KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awareness (2023.findings-emnlp)

Challenge:	Pre-trained language models have shown impressive performance in various language tasks, but are prone to spurious correlations and illusory information.
Approach:	They propose to use pre-trained language models to justify decisions with formalized, coherent reasoning chains.
Outcome:	The proposed strategies improve coherence of rationalizations yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP).

Copied to clipboard

Yichuan Li, Jialong Han, Kyumin Lee, Chengyuan Ma, Benjamin Yao, Xiaohu Liu

ParaMac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms (2022.findings-emnlp)

Challenge:	Pre-trained language models (PLMs) have shown their superiority by pre-training on unstructured text corpus and then fine-tuning on downstream tasks.
Approach:	They propose a Knowledge-Enhanced Pre-trained LanguagE model with Topic entity awareness that incorporates the interactions between tokens and mentioned entities in pre-training.
Outcome:	The proposed model incorporates the interactions between tokens and mentioned entities in pre-training and is more effective on entity-centric tasks.

Copied to clipboard

Jinxin Liu, Jiaxin Shi, Ji Qi, Lei Hou, Juanzi Li, Qi Tian

Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023.findings-acl)

Challenge:	Existing unsupervised methods for paraphrase generation are weak in semantic equivalence or expression diversity.
Approach:	They propose a framework for unsupervised paraphrase generation that employs multi-aspect equivalence constraints and multi-granularity diversifying mechanisms to achieve good semantic equvalence and expressive diversity.
Outcome:	The proposed framework achieves 9.1% and 3.3% absolute gains over previous SOTA on Quora and MSCOCO and can improve to 18.0% and 4.6% on GLUE.

Copied to clipboard

Xuanjie Fang, Sijie Cheng, Yang Liu, Wei Wang

TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference (2021.naacl-main)

Challenge:	Pre-trained language models (PLMs) have shown strong potential in various downstream tasks.
Approach:	They propose to model adversarial attack task as a sequential decision-making problem where the whole attack process is sequential with two decision- making problems, i.e., word finder and word substitution.
Outcome:	The proposed approach achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT.

Copied to clipboard

Deming Ye, Yankai Lin, Yufei Huang, Maosong Sun

UOR: Universal Backdoor Attacks on Pre-trained Language Models (2024.findings-acl)

Challenge:	Existing pre-trained language models (PLMs) are expensive in inference, making them impractical in resource-limited real-world applications.
Approach:	They propose a dynamic token reduction approach to accelerate PLMs' inference by adapting the layer number of each token to avoid redundant calculation.
Outcome:	The proposed approach speeds up BERT by 2-5 times and improves performance in long-text tasks with less computation.

Copied to clipboard

Wei Du, Peixuan Li, Haodong Zhao, Tianjie Ju, Ge Ren, Gongshen Liu

Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages (2025.findings-naacl)

Challenge:	Existing methods to attack pre-trained language models rely on manual selection of triggers and backdoor representations.
Approach:	They propose a backdoor attack method that turns manual selection into automatic optimization . they propose to use poisoned contrastive learning to learn more uniform backdoor representations .
Outcome:	The proposed method achieves better attack performance on text classification tasks compared to manual methods.

Copied to clipboard

Poulami Ghosh, Raj Dabre, Pushpak Bhattacharyya

Enhancing Rumor Detection Methods with Propagation Structure Infused Language Model (2025.coling-main)

Challenge:	Existing studies do not focus on linguistically grounded attacks, but pre-trained models are susceptible to these perturbations.
Approach:	They propose to examine whether pre-trained language models are agnostic to linguistically grounded attacks . they find that PLMs are less susceptible to linguistic perturbations than non-linguistic ones .
Outcome:	The proposed model is agnostic to linguistically grounded attacks, but is less susceptible to linguist attacks than non-linguistic models.

Copied to clipboard

Chaoqun Cui, Siyuan Li, Kunkun Ma, Caiyan Jia

DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain (2024.lrec-main)

Challenge:	Pretrained Language Models excel in various Natural Language Processing tasks, but performance on social media applications like rumor detection remains suboptimal.
Approach:	They propose a pretraining strategy to infuse information from propagation structures into pretrained language models to capture interactions of stance and sentiment crucial for rumor detection.
Outcome:	The proposed model outperforms existing methods on social media applications and significantly improves rumor detection performance.

Copied to clipboard

Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacome Constant Dit Beaufils, Natalia Grabar, Béatrice Daille, Solen Quiniou, Emmanuel Morin, Pierre-Antoine Gourraud, Richard Dufour

Soft Syntactic Reinforcement for Neural Event Extraction (2025.naacl-long)

Challenge:	Existing benchmarks for pre-trained language models are limited to only a few languages . a limited number of tasks are evaluated on non-standardized protocols .
Approach:	They propose to aggregate diverse downstream tasks into a benchmark to assess PLMs' qualities . they evaluate 8 pre-trained masked language models on general and biomedical-specific data .
Outcome:	The proposed benchmark assesses pre-trained language models on 20 diversified tasks.

Copied to clipboard

Anran Hao, Jian Su, Shuo Sun, Teo Yong Sen

CLEVE: Contrastive Pre-training for Event Extraction (2021.acl-long)

Challenge:	Recent event extraction methods rely on pre-trained language models but still suffer from errors due to a lack of syntactic knowledge.
Approach:	They propose a method to incorporate syntactic information into PLM-based models for event extraction (EE) this method uses a standard dependency corpus to select syntax-related dimensions of the model's representation.
Outcome:	The proposed method outperforms baseline models and existing syntactic reinforcement methods on sentence-level and document-level EE benchmark datasets.

Copied to clipboard

Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, Jie Zhou

Introducing Graph Context into Language Models through Parameter-Efficient Fine-Tuning for Lexical Relation Mining (2025.acl-long)

Challenge:	Existing EE methods do not model event characteristics from large unsupervised data.
Approach:	They propose a contrastive pre-training framework for event extraction to better learn event knowledge from large unsupervised data and their semantic structures.
Outcome:	The proposed framework improves on ACE 2005 and MAVEN datasets on event extraction tasks.

Copied to clipboard

Jingwen Sun, Zhiyi Tian, Yu He, Jingwei Sun, Guangzhong Sun

SALMON: A Structure-Aware Language Model with logicality and densification strategy for Temporal Knowledge Graph Reasoning (2024.findings-emnlp)

Challenge:	Pre-trained language models can effectively mine lexical relations between word pairs . however, graph features and semantic knowledge of pre-tried models are lacking in the task.
Approach:	They propose a parameter-efficient fine-tuning method which integrates graph features and semantic representations for lexical relation classification and lexic entailment tasks.
Outcome:	The proposed method integrates graph features and semantic representations for lexical relation mining tasks.

Copied to clipboard

Fu Zhang, Jinghao Lin, Jingwei Cheng

Benchmarking Large Language Model Capabilities for Conditional Generation (2023.acl-long)

Challenge:	Temporal knowledge graph reasoning (TKGR) is a crucial task that involves reasoning at known timestamps to complete the future facts.
Approach:	They propose a temporal knowledge graph reasoning model with logicality and densification strategy that captures temporal evolving pattern and structural information in TKGs.
Outcome:	The proposed model outperforms the state-of-the-art models and is based on a structure-aware language model with logicality and densification strategy.

Copied to clipboard

Joshua Maynez, Priyanka Agrawal, Sebastian Gehrmann

Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)

Challenge:	Autoregressive and pre-trained large language models have shifted the field from application-specific to generation-based approaches.
Approach:	They propose to adapt existing application-specific generation benchmarks to pre-trained large language models to better suit different tasks.
Outcome:	The proposed models differ in their applicability to different data regimes and their generalization to multiple languages.

Copied to clipboard

Wenxuan Lu, Songhao Jiang, Wang Yijing, Tianning Zang

Enhance Robustness of Language Models against Variation Attack through Graph Integration (2024.lrec-main)

Challenge:	Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities.
Approach:	They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings .
Outcome:	The proposed method improves performance on a T5 model and in decoder-only models .

Copied to clipboard

Zi Xiong, Lizhi Qing, Yangyang Kang, Jiawei Liu, Hongsong Li, Changlong Sun, Xiaozhong Liu, Wei Lu

Enhancing Court View Generation with Knowledge Injection and Guidance (2024.lrec-main)

Challenge:	Pre-trained language models (PLMs) are used in many NLP applications but their vulnerability to adversarial attacks can lead to false or misleading information being distributed.
Approach:	They propose a method to incorporate a Chinese character variation graph into pre-trained language models to increase their robustness against character variation attacks in Chinese content.
Outcome:	The proposed method outperforms existing language models in combating adversarial attacks in Chinese content.

Copied to clipboard

Ang Li, Yiquan Wu, Yifei Liu, Kun Kuang, Fei Wu, Ming Cai

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023.acl-long)

Challenge:	Existing methods of natural language generation (NLG) rely on the extensive parameters of pretrained language models (PLMs) but their effectiveness may be compromised by insufficient domain-specific knowledge.
Approach:	They propose a knowledge-injected prompt encoder to incorporate domain knowledge during the training stage, thereby reducing computational overhead.
Outcome:	The proposed approach outperforms established baselines on real-world data in responsivity of claims and in the ability to transfer domain knowledge.

Copied to clipboard

Jian Yang, Shuming Ma, Li Dong, Shaohan Huang, Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, Zhoujun Li

Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment (2023.findings-emnlp)

Challenge:	Existing pre-training methods underutilize the benefits of language understanding for generation.
Approach:	They propose a GAN-style model for encoder-decoder pre-training with an auxiliary discriminator.
Outcome:	The proposed model outperforms existing pre-trained models and achieves state-of-the-art performance.

Copied to clipboard

Boyang Xue, Weichao Wang, Hongru Wang, Fei Mi, Rui Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

Learning to Imagine: Visually-Augmented Natural Language Generation (2023.acl-long)

Challenge:	Experimental results show that pretrained language models generate inconsistent factual knowledge in many conversational tasks.
Approach:	They propose a method which explicitly introduces extended feedforward networks (FFNs) in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs.
Outcome:	The proposed methods improve the factual expression capability of feedforward networks (FFNs) in knowledge-grounded dialogue systems by knowledge enhancement and alignment respectively.

Copied to clipboard

Tianyi Tang, Yushuo Chen, Yifan Du, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen

Improving Language Models’ Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary (2023.emnlp-main)

Challenge:	Existing methods for natural language generation are pre-trained on text-only corpora, resulting in visual commonsense.
Approach:	They propose a method that makes pre-trained language models learn to imagine for visually-augmented natural language generation.
Outcome:	The proposed method is compatible with Transformer-based architecture.

Copied to clipboard

Myeongjun Jang, Thomas Lukasiewicz

Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)

Challenge:	Existing studies have exploited data augmentation or implemented specialised loss functions to alleviate the inconsistent behaviour issue, but they consume expensive training resources and can only handle a certain consistency type.
Approach:	They propose a method that allows PLMs to capture accurate meaning by learning precise interrelationships between concepts from word-definition pairs in a dictionary.
Outcome:	The proposed method can improve multiple types of consistency and integrate pre-trained knowledge with PLMs’ pre-training knowledge.

Copied to clipboard

Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency

DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023.acl-long)

Challenge:	Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks.
Approach:	They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning .
Outcome:	Using pre-trained language models, we compare three options on NLP classification tasks and domain shift.

Copied to clipboard

Pei Ke, Fei Huang, Fei Mi, Yasheng Wang, Qun Liu, Xiaoyan Zhu, Minlie Huang

Development of Cognitive Intelligence in Pre-trained Language Models (2024.emnlp-main)

Challenge:	Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
Approach:	They propose a metric that evaluates natural language generation tasks as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models without training on evaluation datasets.
Outcome:	The proposed metric achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which exhibits strong dimension-level / task-level generalization ability and interpretability.

Copied to clipboard

Raj Shah, Khushi Bhardwaj, Sashank Varma

Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (2023.findings-acl)

Challenge:	Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). Prior research into emergental cognitive abilities of PLMs has been path-independent to model training.
Approach:	They use four task categories to examine the alignment of ten popular families of PLMs and evaluate their performance to the developmental trajectories of children's thinking.
Outcome:	The results show that the models are more aligned to children's thinking than previous studies.

Copied to clipboard

Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, Greg Ver Steeg

SocioProbe: What, When, and Where Language Models Learn about Sociodemographics (2022.emnlp-main)

Challenge:	Parameter-efficient tuning (PET) methods fit pre-trained language models to downstream tasks by computing a small compressed update for a subset of parameter updates or appending and fine-tuning a few new parameters to the pre-train network.
Approach:	They propose a parameter-efficient tuning method that prunes PET architectures using structured pruning.
Outcome:	The proposed method performs well in practice but could be improved via NAS . it can be used to fine-tune large pre-trained language models on multiple tasks .

Copied to clipboard

Anne Lauscher, Federico Bianchi, Samuel R. Bowman, Dirk Hovy

Can Pre-trained Language Models Interpret Similes as Smart as Human? (2022.acl-long)

Challenge:	Pre-trained language models have outperformed other models on a wide range of tasks . however, there is still little understanding of their knowledge of higher-level aspects of language .
Approach:	They investigate whether pre-trained language models have knowledge of sociodemographics . they use traditional probing techniques to probe the knowledge of single-GPU PLMs based on multiple English data sets .
Outcome:	The results show that pre-trained language models outperform other models on a wide range of tasks.

Copied to clipboard

Qianyu He, Sijie Cheng, Zhixu Li, Rui Xie, Yanghua Xiao

Context-Tuning: Learning Contextualized Prompts for Natural Language Generation (2022.coling-1)

Challenge:	Simile interpretation is a crucial task in natural language processing.
Approach:	They propose a task to let PLMs infer the shared properties of similes by probing textual corpora and human-designed questions.
Outcome:	The proposed task outperforms pre-trained language models on simile interpretation tasks while still underperforming humans.

Copied to clipboard

Tianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen

MVP: Multi-task Supervised Pre-training for Natural Language Generation (2023.findings-acl)

Challenge:	Recent studies have shown that pretrained language models (PLMs) lack sufficient consideration of input semantics to generate natural language.
Approach:	They propose a continuous prompting approach to fine-tune PLMs for natural language generation by modeling an inverse generation process from output to input.
Outcome:	The proposed method fine-tunes only 0.12% of the parameters while maintaining good performance.

Copied to clipboard

Tianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen

Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension (2024.emnlp-main)

Challenge:	Pre-trained language models (PLMs) have achieved remarkable success in natural language generation tasks.
Approach:	They propose to use a large-scale natural language generation corpus to pre-train a text generation model MVP in a supervised manner.
Outcome:	The proposed model outperforms BART and Flan-T5 on 13 out of 17 datasets and outperformed BART by 9.3% and FlaN-T5.

Copied to clipboard

Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023.emnlp-main)

Challenge:	Extractive Machine Reading Comprehension (MRC) is a challenging field in the field of Natural Language Processing.
Approach:	They propose a Question-Attended Span Extraction module to address the limitations of generative approaches for extractive machine reading comprehension (MRC) . module significantly enhances performance of pre-trained generative language models, enabling them to surpass the extractive capabilities of advanced Large Language Models (LLMs)
Outcome:	The QASE module surpasses state-of-the-art models in few-shot settings.

Copied to clipboard

Ahmed Alajrami, Katerina Margatina, Nikolaos Aletras

Dialog-Post: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training (2023.acl-long)

Challenge:	Pre-trained language models capture semantic and syntactic information, but no study has examined how information loss in input token characters affects their performance.
Approach:	They address this gap by pre-training language models using small subsets of token characters.
Outcome:	The proposed model retains 90% and 77% of the full-token model in standard NLU benchmarks and probing tasks even under extreme settings.

Copied to clipboard

Zhenyu Zhang, Lei Shen, Yuming Zhao, Meng Chen, Xiaodong He

Dialogue Systems & Conversational Agents

Challenge:	a new method for dialogue representation and understanding is proposed . pre-trained language models (PLMs) are inappropriate for dialogue understanding tasks .
Approach:	They propose a method that trains pre-trained language models to fit dialogues . they use a hierarchical segment-wise self-attention network to model dialogues more comprehensively .
Outcome:	The proposed method outperforms existing models and achieves a 3.3% improvement on average.

DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking (2024.findings-emnlp)

Copied to clipboard

Devrim Çavuşoğlu, Seçil Şen, Ulaş Sert

PPT: Pre-trained Prompt Tuning for Few-shot Learning (2022.acl-long)

Challenge:	Multiple-choice cloze tests are a prevalent form of assessment that evaluates students' comprehension and inference abilities.
Approach:	They propose a framework for distractor generation using readily available pre-trained language models . human evaluations confirm that their approach produces more effective distractors .
Outcome:	The proposed framework outperforms existing methods without training or fine-tuning human evaluations confirm it.

Copied to clipboard

Yuxian Gu, Xu Han, Zhiyuan Liu, Minlie Huang

ACENet: Attention Guided Commonsense Reasoning on Hybrid Knowledge Graph (2022.emnlp-main)

Challenge:	Prompt tuning for pre-trained language models has shown remarkable performance . however, prompt tuning is still not fully explored .
Approach:	They propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization.
Outcome:	The proposed framework outperforms full-model tuning under full-data and few-shot learning settings.

Copied to clipboard

Chuzhan Hao, Minghui Xie, Peng Zhang

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings (2024.lrec-main)

Challenge:	Existing approaches estimate plausibility of candidate choices separately based on their respective KGs, without considering the interference among different choices.
Approach:	They propose an Attention guided Commonsense rEasoning Network to integrate hybrid knowledge into the neural network.
Outcome:	The proposed model outperforms existing methods on CommonsenseQA and OpenbookQA datasets and shows significant performance gains.

Copied to clipboard

Gaifan Zhang, Yi Zhou, Danushka Bollegala

Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)

Challenge:	Sentence embeddings produced by pretrained language models are high dimensional (ca. 1024-4096) this is problematic when representing large numbers of sentences in memory- or compute-constrained devices.
Approach:	They propose to use Principal Component Analysis to reduce the dimensionality of sentence embeddings produced by pretrained language models to reduce their complexity.
Outcome:	The proposed methods reduce the dimensionality of sentence embeddings by 50% without incurring significant loss in performance in multiple downstream tasks.

Copied to clipboard

Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang

Structural Bias for Aspect Sentiment Triplet Extraction (2022.coling-1)

Challenge:	Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models.
Approach:	They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt.
Outcome:	The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets.

Copied to clipboard

Chen Zhang, Lei Ren, Fang Ma, Jingang Wang, Wei Wu, Dawei Song

Sentiment Analysis

Challenge:	Existing structural bias adapters for aspect sentiment triplet extraction are under-confident . a large-scale dataset for ASTE shows the adapter is effective and efficient to a larger scale.
Approach:	They propose to use a structural adapter to integrate structural bias into pretrained language models . they propose to add a relative position structure in place of the syntactic dependency structure .
Outcome:	The proposed adapter achieves state-of-the-art performance over strong baselines, but with a light parameter demand and low latency.

MetaPro 2.0: Computational Metaphor Processing on the Effectiveness of Anomalous Language Modeling (2024.findings-acl)

Copied to clipboard

Rui Mao, Kai He, Claudia Ong, Qian Liu, Erik Cambria

Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks (2022.coling-1)

Challenge:	Existing methods for metaphor interpretation are slow due to lack of annotated datasets and effective pre-trained language models.
Approach:	They propose a large annotated dataset and a PLM for the metaphor interpretation task.
Outcome:	The proposed method improves on metaphor identification and interpretation with comparable baselines on the new dataset.

Copied to clipboard

Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuanjing Huang, Rui Xie, Wei Wu

Mitigate Extrinsic Social Bias in Pre-trained Language Models via Continuous Prompts Adjustment (2024.emnlp-main)

Challenge:	Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications.
Approach:	They propose a framework that re-uses existing parameter-efficient methods with a unified classifier.
Outcome:	The proposed framework improves the efficiency of existing parameter-efficient methods with a unified classifier.

Copied to clipboard

Yiwei Dai, Hengrui Gu, Ying Wang, Xin Wang

Making Pre-trained Language Models both Task-solvers and Self-calibrators (2023.findings-acl)

Challenge:	Existing methods of extrinsic bias mitigation rely on manual word lists for sensitive groups . however, these word lists are limited by length and scope, resulting in poor performance.
Approach:	They propose a method which generates continuous token lists from the entire vocabulary space and uses them to bridge the gap between outputs and targets in fairness learning process.
Outcome:	The proposed method outperforms baseline methods on three NLU tasks.

Copied to clipboard

Yangyi Chen, Xingyao Wang, Heng Ji

FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models (2023.findings-acl)

Challenge:	Existing work shows that pre-trained language models can be effective for high-stake applications, but they become overconfident in their wrong predictions.
Approach:	They propose to use extra data to train pre-trained language models to effectively utilize training samples to make them both task-solvers and self-calibrators.
Outcome:	The proposed method can be used in three downstream applications, including selective classification, adversarial defense, and model cascading.

Copied to clipboard

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, Zenglin Xu

Towards Better Representations for Multi-Label Text Classification with Multi-granularity Information (2023.findings-emnlp)

Challenge:	Existing research on federated learning (FL) for pre-trained language models (PLMs) with increasing concerns about data privacy, enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning PLMs.
Approach:	They investigate the parameter-efficient tuning of pre-trained language models (PLMs) and develop a federated benchmark for four representative PETuning methods .
Outcome:	The proposed method can defend against privacy attacks and maintain acceptable performance with reducing heavy resource consumption.

Copied to clipboard

Fangfang Li, Puzhen Su, Junwen Duan, Weidong Xiao

Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning (2023.findings-acl)

Challenge:	Existing studies have shown that pre-trained language models generate word frequency-oriented text representations, causing texts with different labels to be closely distributed in a narrow region, which is difficult to classify.
Approach:	They propose a framework to refine the text representation for multi-label text classification using contrastive learning and multi-task learning modules.
Outcome:	The proposed framework improves the quality of the representations and yields stable and competitive improvements.

Copied to clipboard

Weize Chen, Xu Han, Yankai Lin, Zhiyuan Liu, Maosong Sun, Jie Zhou

Gender Identity in Pretrained Language Models: An Inclusive Approach to Data Creation and Probing (2024.findings-emnlp)

Challenge:	Existing methods for tuning pre-trained language models ignore the running cost and only optimize the terminal cost.
Approach:	They propose to use stochastic bridges to regularize intermediate states and use regularization as running cost of PETs.
Outcome:	The proposed methods can be used to tune large pre-trained language models . they can be compared to full-parameter fine-tuning by tuning a small number of parameters .

Copied to clipboard

Urban Knupleš, Agnieszka Falenska, Filip Miletić

Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)

Challenge:	Pretrained language models encode binary gender information of text authors, raising the risk of skewed representations and downstream harms.
Approach:	They use a corpus of YouTube transcripts from transgender, cisgender and non-binary speakers to examine whether pretrained language models encode binary gender information.
Outcome:	The proposed model encodes gender information for all gender identities but to different extents.

Copied to clipboard

Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong

Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models (2023.emnlp-main)

Challenge:	Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications.
Approach:	They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators.
Outcome:	The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction.

Copied to clipboard

Miaoxi Zhu, Qihuang Zhong, Li Shen, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)

Challenge:	Existing zero-shot quantization methods are based on overfitting problem in adversarial learning process, leading to sub-optimal performance.
Approach:	They propose a zero-shot sharpness-aware quantization framework for the quantization of various PLMs by optimizing a minimax problem.
Outcome:	The proposed framework can achieve significant performance gains on discriminative and generative PLMs.

Copied to clipboard

Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli

Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)

Challenge:	Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities.
Approach:	They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance.
Outcome:	The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.

Copied to clipboard

Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2020.acl-main)

Challenge:	Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples.
Approach:	They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses.
Outcome:	The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance.

Copied to clipboard

Nora Kassner, Hinrich Schütze

A Two-Stage Adaptation of Large Language Models for Text Ranking (2024.findings-acl)

Challenge:	Petroni et al. ( 2019) show that PLMs can recall factual knowledge that is part of their training corpus.
Approach:	They propose two new probing tasks that analyze factual knowledge stored in pretrained language models (PLMs) they introduce negated and non-negated cloze questions and add misprimes to clozing questions to mimic human priming methods.
Outcome:	The proposed tasks outperform automatically extracted knowledge bases on QA.

Copied to clipboard

Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang

HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning (2025.findings-acl)

Challenge:	Recent advances in pre-trained language models (PLMs) have significantly improved ranking performance in text ranking tasks.
Approach:	They propose a two-stage progressive paradigm to better adapt LLMs to text ranking by conducting continual pre-training on a large weakly-supervised corpus and performing SFT on high-quality data.
Outcome:	The proposed approach outperforms previous methods on in- and out-domain scenarios.

Copied to clipboard

Hoang Tran Vuong, Tue Le, Tu Vu, Tung Nguyen, Linh Ngo Van, Sang Dinh, Thien Huu Nguyen

Recyclable Tuning for Continual Pre-training (2023.findings-acl)

Challenge:	Recent advances in neural topic models (NTMs) have improved topic quality but still face challenges: weak document-topic alignment, high inference costs due to large pretrained language models, and limited modeling of hierarchical topic structures.
Approach:	They propose a framework that integrates hierarchical clustering and contrastive learning to refine document-topic relationships using compact PLM-based embeddings.
Outcome:	The proposed framework improves topic coherence, topic performance, representation quality and computational efficiency over existing NTMs.

Copied to clipboard

Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou

Efficient Pointwise-Pairwise Learning-to-Rank for News Recommendation (2024.findings-emnlp)

Challenge:	Continual pre-training is the paradigm where pre-trained language models acquire fresh knowledge and gradually get upgraded.
Approach:	They propose to use adapted weights to recycle old PLMs for continual pre-training . they propose to combine initialization and distillation methods to achieve better performance .
Outcome:	The proposed method improves the convergence and performance of the upgraded PLM.

Copied to clipboard

Nithish Kannen, Yao Ma, Gerrit Van Den Burg, Jean Baptiste Faddoul

Improving Sequential Model Editing with Fact Retrieval (2023.findings-emnlp)

Challenge:	Recent work leverages the power of pretrained language models to rank news items . pointwise approaches fail to capture comparative information between items that is more effective for ranking tasks.
Approach:	They propose a framework for PLM-based news recommendation that integrates pointwise relevance prediction and pairwise comparisons in a scalable manner.
Outcome:	The proposed framework outperforms state-of-the-art methods on the MIND and Adressa news recommendation datasets.

Copied to clipboard

Xiaoqi Han, Ru Li, Hongye Tan, Wang Yuanlong, Qinghua Chai, Jeff Pan

Representation Learning with Conditional Information Flow Maximization (2024.acl-long)

Challenge:	Existing methods to fix erroneous knowledge in Pre-trained Language models experience a performance decline when the number of edits increases.
Approach:	They propose a framework that leverages factual information to enhance editing generalization and guide the identification of edits by retrieving related facts from the fact-patch memory.
Outcome:	The proposed framework can improve model generalization and accuracy even with thousands of edits.

Copied to clipboard

Dou Hu, Lingwei Wei, Wei Zhou, Songlin Hu

Learning In-context Learning for Named Entity Recognition (2023.acl-long)

Challenge:	Existing knowledge-theoretic representation learning frameworks are based on the information bottleneck principle, which preserves redundant features irrelevant to the given task.
Approach:	They propose a conditional information flow maximization framework to learn sufficient representations for the input data and target task by maximizing both input-representation and representation-label mutual information.
Outcome:	The proposed framework can extract noise-invariant sufficient representations for the input data and target task.

Copied to clipboard

Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, Le Sun

Improving Knowledge Graph Completion with Structure-Aware Supervised Contrastive Learning (2024.emnlp-main)

Challenge:	Existing methods to recognize entities in text are limited by the diversity of entity types and the lack of high-quality annotations.
Approach:	They propose an in-context learning-based NER approach that can inject in-const NER ability into PLMs and recognize entities of novel types on-the-fly using only a few demonstrative instances.
Outcome:	The proposed method outperforms the PLMs+fine-tuning counterparts on 4 few-shot NER datasets and significantly outperformed the Plms+initialized extractors.

Copied to clipboard

Jiashi Lin, Lifang Wang, Xinyu Lu, Zhongtian Hu, Wei Zhang, Wenxuan Lu

Efficient Cross-Task Prompt Tuning for Few-Shot Conversational Emotion Recognition (2023.findings-emnlp)

Challenge:	Existing contrastive methods focus on individual triples, overlooking the broader structural connectivities and topologies of KGs.
Approach:	They propose a new contrastive learning framework that incorporates four tasks specifically tailored to KG data: Vertex-level CL, Neighbor-level Cl, Path-levelCL, and Relation composition level CL.
Outcome:	The proposed framework achieves SOTA performance under standard supervised and low-resource settings.

Copied to clipboard

Yige Xu, Zhiwei Zeng, Zhiqi Shen

Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022.emnlp-main)

Challenge:	Emotion Recognition in Conversation (ERC) models are often expensive to train and fine-tune .
Approach:	They propose a derivative-free optimization method for few-shot conversational emotion recognition that leverages sharable cross-task knowledge by exploiting external knowledge from other source tasks.
Outcome:	The proposed method improves on few-shot scenarios and zero-shot transfers on five different contextual conversation datasets.

Copied to clipboard

Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, Houfeng Wang

HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection (2025.findings-emnlp)

Challenge:	Existing work focuses on monolingual prompts, but we study multilingual prompt for multilingual models.
Approach:	They propose a model that uses a unified prompt for all languages, called UniPrompt, to alleviate the effort of designing different prompts for multiple languages.
Outcome:	The proposed model outperforms baseline models in the zero-shot cross-lingual setting.

Copied to clipboard

Guang Yang, Yujie Zhu

GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (2023.findings-acl)

Challenge:	Pre-trained language models (PLMs) are increasingly being applied to code-related tasks.
Approach:	They propose a hypergraph-based adapter to capture high-order data correlations in code tokens . they improve hypergraph neural networks and combine it with adapter tuning to propose adapter .
Outcome:	The proposed adapter can encode high-order data correlations and be inserted into PLMs to enhance performance.

Copied to clipboard

Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang

Few-Shot Named Entity Recognition: An Empirical Baseline Study (2021.emnlp-main)

Challenge:	Pre-trained language models (PLMs) have improved generalization performance but the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks.
Approach:	They propose to create a benchmark for evaluating out-of-distribution (OOD) generalization in NLP models.
Outcome:	The proposed benchmarks highlight the importance of OOD robustness and provide insights on how to measure it and improve it.

Copied to clipboard

Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, Jiawei Han

DiSec: Mitigating Backdoors in Pre-trained Language Models via Disentanglement of Adversarial Weights for Secure Fine-Tuning (2026.findings-acl)

Challenge:	Existing methods to build named entity recognition systems with limited labeled data are lacking.
Approach:	They propose three orthogonal schemes to build named entity recognition systems when labeled data is limited.
Outcome:	The proposed NER systems outperform existing methods on few-shot and training-free settings.

Copied to clipboard

Sunanda Das, Qinghua Li

Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation (2024.lrec-main)

Challenge:	Existing defenses rely on privileged assumptions, limiting their applicability in realistic settings.
Approach:	They propose a task-agnostic backdoor attack that contaminates pre-trained language models . authors propose auxiliary text purification framework that uses only clean auxiliary data .
Outcome:	The proposed framework suppresses attack success while preserving clean-task utility.

Copied to clipboard

Mateusz Klimaszewski, Piotr Andruszkiewicz, Alexandra Birch

DADA: Distribution-Aware Domain Adaptation of PLMs for Information Retrieval (2024.findings-acl)

Challenge:	Existing approaches to modularity are limited to the case of pre-trained modules in a pre-training language model.
Approach:	They propose a method that allows the transfer of pre-trained PEFT modules between incompatible PLMs without any change in the inference complexity.
Outcome:	The proposed method allows the transfer of modules between incompatible PLMs without any change in the inference complexity.

Copied to clipboard

Dohyeon Lee, Jongyoon Kim, Seung-won Hwang, Joonsuk Park

Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need (2023.emnlp-main)

Challenge:	Pre-trained language models struggle with out-of-domain data due to distribution shifts . generative domain adaptation (DA) methods have been proposed to address these issues .
Approach:	They propose a distribution-aware domain adaptation method to address distribution shifts in domains . they use observation-level feedback and observation- level feedback to adapt to the target domain .
Outcome:	The proposed method adapts to the domain distribution knowledge at the level of a single document and the corpus and expands document representation to unseen gold query terms using domain and observation feedback.

Copied to clipboard

Jinxuan Wu, Wenhan Chao, Xian Zhou, Zhunchen Luo

Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer (2025.findings-acl)

Challenge:	a scientific claim verification requires thorough examination and assessment to ascertain its validity . attention architectures and pre-trained language models fail to establish a comprehensive chain of causal inference .
Approach:	They propose a qualitative causal structure-based graph neural network model to facilitate causal reasoning across relevant causally-potent factors.
Outcome:	The proposed model outperforms state-of-the-art models by incorporating semantic features . the proposed model is based on a qualitative causal structure .

Copied to clipboard

Seungyoon Lee, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim

Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) are increasingly incorporating multilingual capabilities, fueling the demand to transfer them into target language-specific models.
Approach:	They propose a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models to transmit deep representational strengths to LLMs.
Outcome:	The proposed technique outperforms existing methods in cross-lingual understanding setups and achieves faster convergence and lower loss during language adaptation.

Copied to clipboard

Song Wang, Zhen Tan, Ruocheng Guo, Jundong Li

Language-Agnostic Bias Detection in Language Models with Bias Probing (2023.findings-emnlp)

Challenge:	Pretrained Language Models (PLMs) are advanced but data labels are noisy due to the complex annotation process.
Approach:	They propose a framework for fine-tuning PLMs using noisy labels that incorporates guidance from Large Language Models like ChatGPT.
Outcome:	Experiments on synthetic and real-world noisy datasets show that the proposed framework outperforms the state-of-the-art framework.

Copied to clipboard

Abdullatif Köksal, Omer Yalcin, Ahmet Akbiyik, M. Kilavuz, Anna Korhonen, Hinrich Schuetze

Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)

Challenge:	Pretrained language models (PLMs) contain strong social biases, which are difficult to quantify because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input.
Approach:	They propose a bias probing technique called LABDet to evaluate social bias in pretrained language models with a language-agnostic method.
Outcome:	The proposed method “surfaces” nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection.

Copied to clipboard

Kai Zheng, Qingfeng Sun, Yaming Yang, Tengchao Lv, Yeyong Pi, Changlin Zhao, Fei Xu, Qi Zhang

Know-Adapter: Towards Knowledge-Aware Parameter-Efficient Transfer Learning for Few-shot Named Entity Recognition (2024.lrec-main)

Challenge:	Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts.
Approach:	They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models.
Outcome:	The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks.

Copied to clipboard

Binling Nie, Yiming Shao, Yigang Wang

Exploring Lottery Prompts for Pre-trained Language Models (2023.acl-long)

Challenge:	Named entity recognition (NER) is a fundamental task in natural language processing.
Approach:	They propose a knowledgeable adapter to incorporate structure and semantic knowledge of knowledge graphs into PLMs for few-shot NER.
Outcome:	The proposed adapter improves the quality of retrieved information by adding explicit knowledge from external sources to PEFTs.

Copied to clipboard

Yulin Chen, Ning Ding, Xiaobin Wang, Shengding Hu, Haitao Zheng, Zhiyuan Liu, Pengjun Xie

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models (2024.findings-emnlp)

Challenge:	Existing approaches to optimize pre-trained language models are expensive and slow to scale.
Approach:	They propose to search for instance-level lottery prompts and generalize them to unseen data . they validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM .
Outcome:	The proposed method can achieve comparable results with other gradient-free and optimization-free baselines.

Copied to clipboard

Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly

A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024.emnlp-main)

Challenge:	Current vocabulary adaptation approaches append the target domainspecific vocabulary (V DOMAIN) at the end of the PLM vocabulary.
Approach:	They propose a vocabulary adaptation scheme that appends a target domain-specific vocabulary (V DOMAIN) at the end of the PLM vocabulary.
Outcome:	The proposed approach improves by 3.57% (in terms of accuracy) and 1.87% (royal-L) over various classification and summarization tasks.

Copied to clipboard

Zihao Li, Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann

ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning (2024.findings-emnlp)

Challenge:	Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community.
Approach:	They propose to compare multilingual pretraining objectives in a controlled methodological environment with multilingual models.
Outcome:	The proposed model outperforms existing models in 6 languages and demonstrates that multilingual translation is an effective pretraining objective under the right conditions.

Copied to clipboard

Yu-Chen Lin, Wei-Hua Li, Jun-cheng Chen, Chu-Song Chen

Leveraging Linguistically Enhanced Embeddings for Open Information Extraction (2024.lrec-main)

Challenge:	Prompt Tuning has been a popular fine-tuning method for large-scale pretrained language models.
Approach:	They propose a method that allows all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a number of adaptive weights.
Outcome:	The proposed method achieves superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs.

Copied to clipboard

Fauzan Nayeem Farooqui, Thanmay Jayakumar, Pulkit Mathur, Mansi A. Radke

IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method (2024.emnlp-main)

Challenge:	Open Information Extraction (OIE) is a structure prediction task in NLP that aims to extract structured n-ary tuples from free text.
Approach:	They propose to leverage linguistic features with a Seq2Seq PLM for OIE to improve performance.
Outcome:	The proposed methods give any neural OIE architecture the key performance boost from both PLMs and linguistic features in one go.

Copied to clipboard

MiHyeon Kim, Juhyoung Park, YoungBin Kim

Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023.emnlp-main)

Challenge:	Pre-trained Language Models (PLMs) have shown remarkable performance on diverse NLP tasks through pre-training and fine-tuning.
Approach:	They propose a numerically robust IM-connection incorporating a layer of BERT as a solution of Ordinary Differential Equations (ODEs) . Experimental results validate the robustness of IM BERT under various conditions.
Outcome:	The proposed model outperforms the existing model on the adversarial GLUE dataset by 5.9%p on low-resource scenarios.

Copied to clipboard

Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun

The Overlooked Repetitive Lengthening Form in Sentiment Analysis (2024.findings-emnlp)

Challenge:	Parameter-efficient tuning (PET) methods can drive large pre-trained language models by training only minimal parameters.
Approach:	They propose a parameter-efficient tuning method that is compatible with a tunable module and uses a random number generator to optimize fewer table parameters.
Outcome:	The proposed method is compatible with a tunable module and tested on 11 NLP tasks.

Copied to clipboard

Lei Wang, Eduard Dragut

HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024.findings-emnlp)

Challenge:	Existing studies have focused on emphatic and expressive language models with informal styles, such as memes and emojis.
Approach:	They propose a two-stage Explainable Instruction Tuning framework that can improve LLMs' performance and explainability for RLF with limited samples.
Outcome:	The proposed framework can match zero-shot GPT-4 in performance but not explainability for RLF with limited samples.

Copied to clipboard

Chuancheng Lv, Lei Li, Shitou Zhang, Gang Chen, Fanchao Qi, Ningyu Zhang, Hai-Tao Zheng

mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (2024.lrec-main)

Challenge:	Existing approaches to adapt pre-trained language models (PLMs) to emerging tasks are costly and inefficient.
Approach:	They propose a meta-network that generates task-specific weights without any optimization.
Outcome:	The proposed approach has flexible generalization ability and superior performance over hypenetworks.

Copied to clipboard

Christophe Servan, Sahar Ghannay, Sophie Rosset

Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024.findings-emnlp)

Challenge:	Existing studies on the ethical and ecological impact of pre-trained language models raise questions about the temporal, financial, and environmental aspects of such models.
Approach:	They propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLMs.
Outcome:	The proposed model is compared with classical multilingual models and is ethically virtuous.

Copied to clipboard

Zeren Shui, Petros Karypis, Daniel Karls, Mingjian Wen, Saurav Manchanda, Ellad Tadmor, George Karypis

Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization (2023.emnlp-main)

Challenge:	Prior research has shown that pretrained language models (PLMs) can achieve state-of-the-art performance on CIC benchmarks.
Approach:	They propose a multi-task learning framework that fine-tunes pretrained language models on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals.
Outcome:	The proposed framework outperforms current state-of-the-art models on small datasets while aligning with the best-performing model on a large dataset.

Copied to clipboard

Chi Cheang, Hou Chan, Derek Wong, Xuebo Liu, Zhaocong Li, Yanming Sun, Shudong Liu, Lidia Chao

A Causal View of Entity Bias in (Large) Language Models (2023.findings-emnlp)

Challenge:	Existing summarization benchmarks overlap in time with pre-training corpora and fine-tuning datasets.
Approach:	They propose a temporal generalization benchmark that contains data samples from 2010 to 2022 to understand the temporal ability of abstractive summarization models.
Outcome:	The proposed benchmark analyzes data samples from 2010 to 2022 to understand the temporal generalization ability of abstractive summarization models.

Copied to clipboard

Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, Muhao Chen

A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023.emnlp-main)

Challenge:	Entity bias affects pretrained (large) language models, causing them to rely on (biased) parametric knowledge to make unfaithful predictions.
Approach:	They propose a structured causal model whose parameters are easier to estimate . they propose to perturb the original entity with neighboring entities .
Outcome:	The proposed model reduces biasing information pertaining to the original entity while still preserving sufficient semantic information from similar entities.

Copied to clipboard

Anthony Liu, Lajanugen Logeswaran, Sungryull Sohn, Honglak Lee

Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)

Challenge:	Recent work uses pre-trained language models to reason about plans from text instructions in embodied visual environments.
Approach:	They propose to use pre-trained language models to reason about plan sequences from text instructions in embodied visual environments.
Outcome:	The proposed approach outperforms previous approaches on the ALFWorld and VirtualHome benchmarks.

Copied to clipboard

Paul Youssef, Osman Koraş, Meijie Li, Jörg Schlötterer, Christin Seifert

Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation? (2024.emnlp-main)

Challenge:	Pre-trained language models are trained on vast unlabeled data, rich in world knowledge.
Approach:	They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases .
Outcome:	The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work.

Copied to clipboard

Wataru Hashimoto, Hidetaka Kamigaito, Taro Watanabe

On Leveraging Encoder-only Pre-trained Language Models for Effective Keyphrase Generation (2024.lrec-main)

Challenge:	Named Entity Recognition (NER) is a key task in NLP to find mentions of named entities and classify them into predefined categories.
Approach:	They investigated the impact of data augmentation on confidence calibration and uncertainty estimation in Named Entity Recognition (NER) tasks.
Outcome:	The data augmentation improves calibration and uncertainty in cross-genre and cross-lingual setting, especially in-domain setting.

Copied to clipboard

Di Wu, Wasi U. Ahmad, Kai-Wei Chang

On the Relationship between Skill Neurons and Robustness in Prompt Tuning (2024.lrec-main)

Challenge:	a new study examines the use of encoder-only pre-trained language models in keyphrase generation (KPG) keyphrases are phrases that condense salient information of a document.
Approach:	They propose to use encoder-only pre-trained language models in keyphrase generation . they also examine optimal architectural decisions for employing encoder only PLMs in KPG .
Outcome:	The proposed model outperforms general-domain seq2seq models in keyphrase generation.

Copied to clipboard

Leon Ackermann, Xenia Isabel Ohmer

Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding (2024.emnlp-main)

Challenge:	Prompt Tuning is a parameter-efficient finetuning method for pre-trained large language models (PLMs).
Approach:	They propose to use RoBERTa to fine tune pre-trained large language models by finetuning only a small set of parameters to adjust for downstream tasks.
Outcome:	The proposed method activates specific neurons in the transformer’s feed-forward networks that are highly predictive and selective for the given task.

Copied to clipboard

YeonJoon Jung, Jaeseong Lee, Seungtaek Choi, Dohyeon Lee, Minsoo Kim, Seung-won Hwang

Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models (2024.lrec-main)

Challenge:	Existing methods to increase the robustness of pre-trained language models (PLMs) against unseen ASR systems produce noisy inputs for SLU models, which can significantly degrade their performance.
Approach:	They propose to introduce ASR-plausible noises into pre-trained language models by cutting off the non-causal effect of noises.
Outcome:	The proposed method improves the robustness and generalizability of SLU models against unseen ASR systems by cutting off the non-causal effect of noises.

Copied to clipboard

Pingjie Wang, Hongcheng Liu, Yanfeng Wang, Yu Wang

Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers (2024.emnlp-main)

Challenge:	Structured pruning is an effective technique for compressing pre-trained language models (PLMs), but it requires retraining, leading to additional computational overhead.
Approach:	They propose a task-specific pruning framework that prunes redundant modules of pre-trained language models before fine-tuning them.
Outcome:	The proposed pruning framework achieves higher performance on GLUE, SQUAD, WikiText-2, Wik-103, and PTB datasets while reducing the time required for fine-tuning.

Copied to clipboard

Tuc Nguyen, Thai Le

Representation Degeneration Problem in Prompt-based Models for Natural Language Understanding (2024.lrec-main)

Challenge:	Existing studies show that augmenting the training data of pre-trained language models with parametric fine-tuning methods can enhance their robustness under adversarial attacks.
Approach:	They propose an approach that fine-tunes PLMs with adapters and adversarial augmentation via mixup to leverage existing knowledge from a set of pre-known attacks.
Outcome:	The proposed approach achieves best trade-off between training efficiency and robustness under adversarial attacks compared to baselines on five downstream tasks across six varied black-box attacks and 2 PLMs.

Copied to clipboard

Qingyan Zhao, Ruifang He, Jinpeng Zhang, Chang Liu, Bo Wang

Dense Retrieval with Quantity Comparison Intent (2025.findings-acl)

Challenge:	Prompt-based fine-tuning (PF) models have shown improved performance on few-shot natural language understanding benchmarks.
Approach:	They propose a framework to alleviate anisotropy in the embedding space by aligning with pre-trained language models' training objective.
Outcome:	The proposed method outperforms mainstream methods on many NLU benchmarks.

Copied to clipboard

Prayas Agrawal, Nandeesh Kumar K M, Muthusamy Chelliah, Surender Kumar, Soumen Chakrabarti

Revisiting the Self-Consistency Challenges in Multi-Choice Question Formats for Large Language Model Evaluation (2024.lrec-main)

Challenge:	Existing sparse and dense retrieval systems fragment numerals and units that express quantities in arbitrary ways.
Approach:	They propose a dense retrieval system built around a density multi-vector index . they propose eliciting and exploiting quantities and associated comparison intents .
Outcome:	The proposed system is faster and more accurate than popular PLMs on two public and one proprietary e-commerce benchmarks.

Copied to clipboard

Wenjie Zhou, Qiang Wang, Mingzhou Xu, Ming Chen, Xiangyu Duan

Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)

Challenge:	Multi-choice questions (MCQs) are a common method for assessing the world knowledge of large language models.
Approach:	They propose three knowledge-equivalent question variants to assess LLMs' world knowledge . they propose option position shuffle, option label replacement, and conversion to a True/False format .
Outcome:	The proposed questions are shuffle, label replacement, and True/False format.

Copied to clipboard

Xudong Lin, Manling Li, Richard Zemel, Heng Ji, Shih-Fu Chang

SentiCSE: A Sentiment-aware Contrastive Sentence Embedding Framework with Sentiment-guided Textual Similarity (2024.lrec-main)

Challenge:	Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining.
Approach:	They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts.
Outcome:	The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings .

Copied to clipboard

Jaemin Kim, Yohan Na, Kangmin Kim, Sang-Rak Lee, Dong-Kyu Chae

Target-Adaptive Consistency Enhanced Prompt-Tuning for Multi-Domain Stance Detection (2024.lrec-main)

Challenge:	Sentiment-aware pre-trained language models (PLMs) demonstrate impressive results in downstream sentiment analysis tasks, but they neglect to evaluate the quality of constructed sentiment representations.
Approach:	They propose a new metric for evaluating the quality of sentiment representations that is based on the degree of equivalence in sentiment polarity between two sentences.
Outcome:	The proposed framework outperforms the existing sentiment-aware models in sentiment analysis tasks.

Copied to clipboard

Shangkang Wang, Li Pan

Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation (2024.lrec-main)

Challenge:	Stance detection is a fundamental task in natural language processing, but it is challenging due to diverse expressions and topics related to the targets from multiple domains.
Approach:	They propose a prompt-tuning method that incorporates target knowledge and prior knowledge to construct target-adaptive verbalizers for diverse domains.
Outcome:	The proposed method outperforms the state-of-the-art methods on nine stance detection datasets from multiple domains.

Copied to clipboard

Francois Meyer, Jan Buys

LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines (2026.findings-acl)

Challenge:	Existing data-to-text models are designed for the linguistic typology of English, but they are not suitable for low-resource languages.
Approach:	They propose a new dataset based on a subset of WebNLG that is agglutinative and low-resource data-to-text.
Outcome:	The proposed model outperforms existing models for isiXhosa and Finnish and fine-tunes machine translation models as the best method overall.

Copied to clipboard

Jiechao Gao, Rohan Kumar Yadav, Yuangang Li, Yuandong Pan, Jie Wang, Ying Liu, Michael Lepech

Explainability & Interpretability in NLP

Challenge:	Pretrained language models (PLMs) provide strong semantic representations but are costly and opaque.
Approach:	They propose a framework that transfers pretrained language models into symbolic form and integrates them into symbolic models.
Outcome:	The proposed framework improves interpretability and accuracy across multiple text classification tasks while remaining fully symbolic and efficient.

WkNER: Enhancing Named Entity Recognition with Word Segmentation Constraints and kNN Retrieval (2024.lrec-main)

Copied to clipboard

Yanchun Li, Senlin Deng, Dongsu Shen, Shujuan Tian, Saiqin Long

From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanations (2026.findings-acl)

Challenge:	Named Entity Recognition (NER) tasks require detecting the span and category of the entity from the text block.
Approach:	They propose a kNN retrieval enhancement algorithm that incorporates word segmentation information to enhance the model’s generalization ability and alleviate the problem of missing entity tokens in prediction.
Outcome:	The proposed method improves the performance of baseline models and achieves better or compared recognition accuracy than previous state-of-the-art models in multiple public Chinese and English datasets.

Copied to clipboard

Pulkit Bansal, Raghvendra Kumar, Shakti Singh, Adam Jatowt, Sriparna Saha