MLM

NeuralQA: A Usable Library for Question Answering (Contextual Query Expansion + BERT) on Large Datasets (2020.emnlp-demos)

Challenge:	Existing methods to train domain expertise for cybersecurity text domains are expensive to train and run.
Approach:	They propose to use pretraining methods to account for non-linguistic elements in cybersecurity texts and evaluate their effectiveness through downstream tasks and probing tasks.
Outcome:	The proposed strategy outperforms the commonly taken approach of replacing NLEs and outperformed other cybersecurity PLMs on most tasks.

Copied to clipboard

Victor Dibia

Question Answering

Challenge:	Existing tools for Question Answering (QA) have challenges that limit their use in practice.
Approach:	They propose a library that integrates with existing infrastructure and offers helpful defaults for QA subtasks.
Outcome:	NeuralQA integrates well with existing infrastructure and offers helpful defaults for QA subtasks.

Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)

Copied to clipboard

Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang

Arabic Synonym BERT-based Adversarial Examples for Text Classification (2024.eacl-srw)

Challenge:	Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks.
Approach:	They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator.
Outcome:	The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings.

Copied to clipboard

Norah Alshahrani, Saied Alshahrani, Esma Wali, Jeanna Matthews

Robustness in NLP

Challenge:	Often, research studies quantifying the impact of adversarial text attacks have been applied only to models trained in English.
Approach:	They propose a word-level study of adversarial text examples in Arabic . they use a synonym attack with a BERT model to assess their robustness .
Outcome:	The proposed attack compares Arabic adversarial examples with their original examples and regains 2% accuracy after training.

How does the pre-training objective affect what large language models learn about linguistic properties? (2022.acl-short)

Copied to clipboard

Ahmed Alajrami, Nikolaos Aletras

Reference Free Domain Adaptation for Translation of Noisy Questions with Question Specific Rewards (2023.findings-emnlp)

Challenge:	Several pre-training objectives have been proposed to pre-train language models . but, to our knowledge, no studies have investigated how different pre- training objectives affect what BERT learns about linguistic properties.
Approach:	They propose to use masked language modeling to pre-train language models . they propose to optimize a mangled language modeling objective to learn linguistic information .
Outcome:	The proposed objectives improve BERT's learning of linguistic properties compared to non-linguistically motivated objectives.

Copied to clipboard

Baban Gain, Ramakrishna Appicharla, Soumya Chennabasavaraj, Nikesh Garera, Asif Ekbal, Muthusamy Chelliah

Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling (2025.naacl-srw)

Challenge:	Creating a synthetic parallel corpus from noisy data is also difficult due to its noisy nature.
Approach:	They propose a training methodology that fine-tunes the NMT system only using source-side data to balance adequacy and fluency.
Outcome:	The proposed method surpasses the MLE-based fine-tuning approach by achieving a 1.9 BLEU improvement.

Copied to clipboard

Samuel Belkadi, Libo Ren, Nicolo Micheletti, Lifeng Han, Goran Nenadic

TeluguNER: Leveraging Multi-Domain Named Entity Recognition with Deep Transformers (2022.acl-srw)

Challenge:	Existing methods to generate medical records using Causal Language Modelling are limited due to privacy concerns.
Approach:	They propose a method for generating medical records using Masked Language Modelling using Causal language models.
Outcome:	The proposed method produces high-quality synthetic data with a re-identification risk of only 3.5% and a patient recall of 96%.

Copied to clipboard

Suma Reddy Duggenpudi, Subba Reddy Oota, Mounika Marreddy, Radhika Mamidi

UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database (2020.aacl-main)

Challenge:	Named Entity Recognition (NER) is a successful and well-researched problem in English due to the availability of resources.
Approach:	They propose to use two annotated NER datasets for the Telugu language . they compare the finetuned Telugus model with the existing model in NER .
Outcome:	The proposed models outperform existing models on a large dataset of 38,363 sentences on telugu and other languages.

Copied to clipboard

Canwen Xu, Tao Ge, Chenliang Li, Furu Wei

Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)

Challenge:	Chinese and Japanese share many characters with similar surface morphology.
Approach:	They propose a Chinese-Japanese pretrained masked language model with a coarse-to-fine training approach to exploit the shared knowledge across the languages.
Outcome:	The proposed model is effective on mono- and cross-lingual Chinese and Japanese tasks.

Copied to clipboard

Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew Leavitt

Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition (2023.emnlp-main)

Challenge:	Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate.
Approach:	They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively.
Outcome:	The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines.

Copied to clipboard

Chenxu Wang, Ping Jian, Mu Huang

On-the-fly Cross-lingual Masking for Multilingual Pre-training (2023.acl-long)

Challenge:	Existing methods for identifying discourse relations without explicit connectives are limited by the availability of annotated data.
Approach:	They propose a method that injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction.
Outcome:	The proposed method achieves outstanding performance against the current state-of-the-art models.

Copied to clipboard

Xi Ai, Bin Fang

Revisiting Pre-Trained Models for Chinese Natural Language Processing (2020.findings-emnlp)

Challenge:	In multilingual pre-training, multilingual models only learn cross-linguality implicitly from isomorphic spaces formed by overlapping different language spaces due to the lack of explicit cross-linguistic forward pass.
Approach:	They propose a dynamic token-wise masking scheme for multilingual pre-training that uses a special token [C]x to replace a random token in the input sentence.
Outcome:	The proposed model improves the performance of UNMT models on De, Ro, Ne En.

Copied to clipboard

Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, Guoping Hu

PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)

Challenge:	Existing pre-trained language models have shown tremendous improvements across various NLP tasks.
Approach:	They propose to revisit Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pretrained model series to the community.
Outcome:	The proposed model improves on RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac).

Copied to clipboard

Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Kezhi Mao

Language-agnostic BERT Sentence Embedding (2022.acl-long)

Challenge:	Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens.
Approach:	They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations.
Outcome:	Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods.

Copied to clipboard

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, Wei Wang

NCPrompt: NSP-Based Prompt Learning and Contrastive Learning for Implicit Discourse Relation Recognition (2024.findings-emnlp)

Challenge:	Existing methods for learning bilingual sentence embeddings are not well explored.
Approach:	They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba.
Outcome:	The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER.

Copied to clipboard

Yuetong Rong, Yijun Mo

Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)

Challenge:	Recent prompt learning methods have demonstrated success in IDRR, but they fail to fully exploit critical semantic features shared among various forms of templates.
Approach:	They propose an NSP-based prompt learning and contrastive learning method for IDRR that transforms the IDRR task into a next sentence prediction task.
Outcome:	The proposed model can be used to classify the discourse relation sense between argument pairs without an explicit connective.

Copied to clipboard

Richeek Das, Sahasra Ranjan, Shreya Pathak, Preethi Jyothi

Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality (2025.acl-long)

Challenge:	Multilingual pretraining models for code-switched inputs are a key component of NLP applications.
Approach:	They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking.
Outcome:	The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques.

Copied to clipboard

Rahul Zalkikar, Kanchan Chandra

Improving Contextual Representation with Gloss Regularized Pre-training (2022.findings-naacl)

Challenge:	Innovative transformer-based language models produce contextually-aware token embeddings, but have been shown to encode unwanted biases for downstream applications.
Approach:	They extend previous work by evaluating social biases introduced after retraining an MLM under the masked language modeling objective and propose proxy functions within an iterative masking experiment to measure the quality of transformer models’ predictions.
Outcome:	The proposed proxy functions within an iterative masking experiment show that all transformer models encode concerning social biases.

Copied to clipboard

Yu Lin, Zhecheng An, Peihao Wu, Zejun Ma

DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog (2022.findings-acl)

Challenge:	Experimental results show that the gloss regularizer module enhances word semantic similarity in pre-training.
Approach:	They propose an auxiliary gloss regularizer module to BERT pre-training to enhance word semantic similarity.
Outcome:	The proposed model improves word similarity in word-level and sentence-level representation.

Copied to clipboard

Chia-Chien Hung, Anne Lauscher, Simone Ponzetto, Goran Glavaš

ExpertPLM: Pre-training Expert Representation for Expert Finding (2022.findings-emnlp)

Challenge:	Recent work shows that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining.
Approach:	They propose a resource-efficient and modular domain specialization by means of domain adapters in which domain knowledge is encoded.
Outcome:	The proposed framework extracts domain-specific terms and then uses them to build DomainCC and DomainReddit resources based on masked language modeling and response selection objectives.

Copied to clipboard

Qiyao Peng, Hongtao Liu

[MASK] Insertion: a robust method for anti-adversarial attacks (2023.findings-eacl)

Challenge:	Existing methods to learn expert representations based on historical answered questions are inadequate.
Approach:	They propose a pre-training and fine-tuning expert finding framework that integrates expert interest and expertise simultaneously.
Outcome:	The proposed framework integrates expert interest and expertise simultaneously and integrates historical answers with the vote score information to model the expert ability.

Copied to clipboard

Xinrong Hu, Ce Xu, Junlong Ma, Zijian Huang, Jie Yang, Yi Guo, Johan Barthelemy

Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? (2022.naacl-main)

Challenge:	Existing studies have focused on adversarial defenses against pretrained language models.
Approach:	They propose an adversarial defensing algorithm that inserts tokens into input sequences . they show an improvement in accuracy between 3.2 and 11.1 absolute points .
Outcome:	The proposed algorithm improves model accuracy on clean and polluted inputs compared with state-of-the-art models .

Copied to clipboard

Xiang Zhou, Shiyue Zhang, Mohit Bansal

Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection (2020.findings-emnlp)

Challenge:	Recent Part-Of-Speech (POS) induction models assume certain independence assumptions that do not hold in real languages.
Approach:	They propose a Masked Part-of-Speech Model (MPoSM) that can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction.
Outcome:	The proposed model can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction.

Copied to clipboard

Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, Serena Villata

Deriving Language Models from Masked Language Models (2023.acl-short)

Challenge:	Recent studies have demonstrated the effectiveness of cross-lingual language model pre-training on NLP tasks.
Approach:	They propose a hybrid emoji-based Masked Language Model to leverage eojis across languages to improve the learning of short text messages.
Outcome:	The proposed model performs better on German, Italian and Spanish.

Copied to clipboard

Lucas Torroba Hennigen, Yoon Kim

SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage (2025.findings-acl)

Challenge:	Masked language models do not define an explicit distribution over language, but they assume that tokens masked out are conditionally independent given the unmasked tokens.
Approach:	They propose to use a set of MLM's unary conditionals to construct a fully-connected Markov random field over the input to deduce an explicit joint distribution from MLMs.
Outcome:	The proposed method outperforms existing Markov random field-based approaches and outperformed the original model's conditionals.

Copied to clipboard

Xiaoning Dong, Wenbo Hu, Wei Xu, Tianxing He

Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution (2020.coling-main)

Challenge:	Existing methods to design sophisticated instructions for the LLM to follow, or rely on multiple iterations, could hinder the performance and efficiency of jailbreaks.
Approach:	They propose a simple assistive task linkage paradigm which masks harmful keywords within malicious queries and uses a masked language model task to encode the semantics of the mangled keywords.
Outcome:	The proposed paradigm can effectively circumvent LLM safeguards and elicit harmful responses.

Copied to clipboard

Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, Alexander Panchenko

FELIX: Flexible Text Editing Through Tagging and Insertion (2020.findings-emnlp)

Challenge:	Lexical substitution is a powerful technology used in various NLP applications . it generates plausible words that can replace a given word in a textual context .
Approach:	They propose to use a large-scale comparative study to compare lexical substitution methods . they compare existing and new methods using word sense induction datasets .
Outcome:	The proposed methods improve competitive results by incorporating information about the target word into the models.

Copied to clipboard

Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, Guillermo Garrido

Syntactic Text Processing

Challenge:	FELIX is efficient in low-resource settings and fast at inference time, while being capable of modeling flexible input-output transformations.
Approach:	They propose a flexible text-editing approach that decomposes a text-generating task into two sub-tasks: tagging and insertion.
Outcome:	The proposed model is efficient in low-resource settings and fast at inference time while being capable of modeling flexible input-output transformations.

Structural Contrastive Pretraining for Cross-Lingual Comprehension (2023.findings-acl)

Copied to clipboard

Nuo Chen, Linjun Shou, Tengtao Song, Ming Gong, Jian Pei, Jianhui Chang, Daxin Jiang, Jia Li

ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2021.naacl-main)

Challenge:	Existing methods to train multilingual language models using pretraining tasks like mask language modeling have yielded promising results on a wide range of downstream tasks.
Approach:	They propose a new task to align the structural words in a parallel sentence, enhancing models’ ability to comprehend cross-lingual representations.
Outcome:	The proposed task improves model's ability to comprehend cross-lingual representations by increasing the frequency of negative pairings.

Copied to clipboard

Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model (2021.acl-long)

Challenge:	Existing methods to model coarse-grained linguistic information do not integrate coarse-gram information into pre-training.
Approach:	They propose an explicitly n-gram masking method to enhance integration of coarse-grained linguistic information into pre-training.
Outcome:	The proposed method outperforms existing models on English and Chinese text corpora and fine-tunes on 19 downstream tasks.

Copied to clipboard

Hongliang Dai, Yangqiu Song, Haixun Wang

Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021.eacl-main)

Challenge:	Existing methods for fine-grained entity typing use weak labels that are automatically generated.
Approach:	They propose to obtain training data by using a BERT Masked Language Model (MLM) given a mention in a sentence, they construct an input for the MLM so it predicts context dependent hypernyms of the mention, which can be used as type labels.
Outcome:	The proposed model improves performance by using type labels generated from a BERT Masked Language Model given a mention in a sentence.

Copied to clipboard

Tianxing He, Bryan McCann, Caiming Xiong, Ehsan Hosseini-Asl

Graph-Based Phonetic Error Correction of Noisy ASR (2026.acl-industry)

Challenge:	Existing calibration methods rescale posterior distributions of classifiers after training.
Approach:	They propose to use a noise contrastive estimation technique to train an energy-based model during finetuning of pretrained text encoders.
Outcome:	The proposed model can reach a better calibration competitive to strong baselines with little or no loss in accuracy.

Copied to clipboard

Pratik Rakesh Singh, Mohammadi Zaki, Aneesh Mukkamla, Pankaj Wasnik

A Two-Stage Framework with Self-Supervised Distillation for Cross-Domain Text Classification (2024.lrec-main)

Challenge:	Automatic speech recognition systems produce residual transcription errors that affect semantically critical tokens.
Approach:	They propose a phonetic-based algorithm that combines phonetic graph modeling with contextual language understanding to improve automatic speech recognition.
Outcome:	The proposed framework decouples phonetic reasoning from contextual semantic selection and improves accuracy.

Copied to clipboard

Yunlong Feng, Bohan Li, Libo Qin, Xiao Xu, Wanxiang Che

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision (2021.emnlp-main)

Challenge:	Existing work on cross-domain text classification relies on domain-invariant features or task-agnostic features.
Approach:	They propose a two-stage framework for cross-domain text classification that leverages or reuses rich labeled data from the source domain and unlabeled data in the target domain.
Outcome:	The proposed framework achieves state-of-the-art on a public cross-domain text classification benchmark.

Copied to clipboard

Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun

HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020.emnlp-main)

Challenge:	Recent state-of-the-art (SOTA) effective neural network methods have been used in Chinese word segmentation (CWS) However, the robustness of the previous neural methods is limited by the large-scale annotated corpus.
Approach:	They propose a self-supervised Chinese word segmentation approach with a straightforward and effective architecture.
Outcome:	The proposed approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness.

Copied to clipboard

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu

Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation? (2023.findings-eacl)

Challenge:	HERO is a framework for large-scale video+language omni-representation learning.
Approach:	They propose a framework for large-scale video+language omni-representation learning that encodes multimodal inputs in a hierarchical structure and uses Masked Language Modeling and Masked Frame Modeling to train models.
Outcome:	The proposed framework achieves state-of-the-art on multiple benchmarks over text-based video/video-moment retrieval, video question answering (QA), Video-and-language Inference and video Captioning tasks across different domains.

Copied to clipboard

Hiroto Tamura, Tosho Hirasawa, Hwichan Kim, Mamoru Komachi

Text Generation

Challenge:	Pre-training masked language models with artificial data has been proven beneficial for several natural language processing tasks, however, it has been less explored for neural machine translation (NMT).
Approach:	They pre-trained masked language models with random sequences and created artificial data mimicking token frequency information from the real world.
Outcome:	The results show that pre-training models with artificial data improves translation performance in low-resource situations.

UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)

Copied to clipboard

Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu

Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)

Challenge:	Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data.
Approach:	They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts.
Outcome:	The proposed model breaks through performance upper bounds of experts without additional annotated data.

Copied to clipboard

Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, Leyu Lin

Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining (2023.findings-eacl)

Challenge:	Existing methods to improve pre-trained language models address information confusion in position encoding and model representations.
Approach:	They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers.
Outcome:	The proposed techniques can improve pre-trained language models on GLUE benchmarks.

Copied to clipboard

Inderjeet Nair, Natwar Modani

Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)

Challenge:	Pretraining large language models has resulted in tremendous performance improvement for many natural language processing tasks.
Approach:	They propose to incorporate pretraining objectives that explicitly exploit domain specific language characteristics into the model.
Outcome:	The proposed objectives target token-level feature representation and incorporate sentence level semantics.

Copied to clipboard

Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, Lei Li

MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning (2026.findings-eacl)

Challenge:	masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations.
Approach:	They propose a representation learning approach that uses embeddings as anchors to model contextual representations.
Outcome:	The proposed model achieves 5x speedup and 1.2 points average improvement over MLM.

Copied to clipboard

Vera Pavlova, Mohammed Makhlouf

To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks (2020.acl-main)

Challenge:	a new framework for domain adaptation of text embedding models addresses the challenges of adapting general-domain text embeds to specialized domains.
Approach:	They propose a framework for domain adaptation of text embedding models that integrates masked supervision and mangled objectives within a unified training pipeline.
Outcome:	The proposed framework improves on high-resource and low-resourced domains while preserving the robustness of the original model.

Copied to clipboard

Sinong Wang, Madian Khabsa, Hao Ma

Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024.naacl-long)

Challenge:	Existing studies on pretraining NLP models with variants of Masked Language Model (MLM) objectives have shown that the number of training samples used in the downstream task is limited.
Approach:	They propose to use MLM objectives to pretrain NLP models with variants of Masked Language Model (MLM) objectives to improve accuracy on downstream tasks.
Outcome:	The proposed model can reach a diminishing return point as the supervised data size increases significantly.

Copied to clipboard

Kraig Tou, Zijun Sun

KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering (2022.emnlp-main)

Challenge:	masked language modeling is widely used as a pretraining component in Vision and language (V+L) but performance on benchmarks has not received the attention it deserves.
Approach:	They propose a curriculum masking scheme that uses a parallel mask selection agent to mask tokens at a frequency proportional to the level of cross modal interaction necessary to reconstruct them.
Outcome:	The proposed method improves relational understanding on a wide range of V+L tasks.

Copied to clipboard

Jianing Wang, Chengyu Wang, Minghui Qiu, Qiuhui Shi, Hongbin Wang, Jun Huang, Ming Gao

Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (2023.eacl-main)

Challenge:	Extractive Question Answering (EQA) is one of the most essential tasks in Machine Reading Comprehension (MRC).
Approach:	They propose a framework that transforms extractive question answering into a non-autoregressive Masked Language Modeling (MLM) generation problem.
Outcome:	The proposed framework outperforms state-of-the-art approaches in few-shot learning scenarios by a large margin.

Copied to clipboard

Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, Miguel Ballesteros

Unsupervised Improvement of Factual Knowledge in Language Models (2023.eacl-main)

Challenge:	Temporal concept drift is a problem of data changing over time.
Approach:	They benchmark 11 pretrained masked language models on a series of tests to evaluate temporal concept drift.
Outcome:	The proposed framework evaluates 11 pretrained masked language models on a series of tests . it aims to reveal how robust an MLM is over time and provide a signal in case it has become outdated .

Copied to clipboard

Nafis Sadeq, Byungkyu Kang, Prarit Lamba, Julian McAuley

Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)

Challenge:	Masked language modeling (MLM) is often dominated by high-frequency words that are sub-optimal for learning factual knowledge.
Approach:	They propose an approach that forces the model to prioritize informative words in a fully unsupervised way.
Outcome:	The proposed approach significantly improves the performance of pretrained language models on factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting.

Copied to clipboard

Alexander Wettig, Tianyu Gao, Zexuan Zhong, Danqi Chen

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)

Challenge:	Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations.
Approach:	They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training.
Outcome:	The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy.

Copied to clipboard

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

Masked Language Models Know Which are Popular: A Simple Ranking Strategy for Commonsense Question Answering (2022.findings-emnlp)

Challenge:	masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say .
Approach:	They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks.
Outcome:	The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models .

Copied to clipboard

Xuan Luo, Chuang Fan, Yice Zhang, Wanguo Jiang, Bing Qin, Ruifeng Xu

HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification (2022.emnlp-main)

Challenge:	Empirical results show that pre-trained language models can improve the typical answer generation of GLMs.
Approach:	They propose a ranking strategy that exploits WordNet to train a ranker that picks out the most popular answers for commonsense questions.
Outcome:	The proposed ranking strategy is tested on a commonsense question answering (QA) dataset and on negative samples from WordNet.

Copied to clipboard

Zihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui, Houfeng Wang

Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)

Challenge:	Hierarchical text classification (HTC) is a multi-label classification problem with a complex label hierarchy.
Approach:	They propose a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label perspective using a dynamic virtual template and label words that take the form of soft prompts to fuse the label hierarchy knowledge.
Outcome:	The proposed method achieves state-of-the-art performance on 3 popular HTC datasets and is adept at handling imbalance and low resource situations.

Copied to clipboard

Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, Nikolaos Aletras

Data Efficient Masked Language Modeling for Vision and Language (2021.findings-emnlp)

Challenge:	Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations.
Approach:	They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective.
Outcome:	Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture.

Copied to clipboard

Yonatan Bitton, Michael Elhadad, Gabriel Stanovsky, Roy Schwartz

Enabling Autoregressive Models to Fill In Masked Tokens (2026.findings-eacl)

Challenge:	Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining.
Approach:	They propose a masking strategy that masks tokens with a 15% probability for text-only data.
Outcome:	The proposed masking strategy outperforms the baseline model on a prompt-based probing task designed to elicit image objects.

Copied to clipboard

Daniel Mingyi Israel, Aditya Grover, Guy Van den Broeck

Exploring Unsupervised Pretraining Objectives for Machine Translation (2021.findings-acl)

Challenge:	Autoregressive (AR) and masked language modeling (MLM) models are incapable of mucked infilling, which is the ability to predict mangled tokens between past and future context.
Approach:	They propose a method that leverages the strengths of autoregressive and masked language modeling to achieve state-of-the-art mucked infilling performance.
Outcome:	The proposed approach outperforms existing methods on masked infilling tasks.

Copied to clipboard

Christos Baziotis, Ivan Titov, Alexandra Birch, Barry Haddow

Token Dropping for Efficient BERT Pretraining (2022.acl-long)

Challenge:	Unsupervised cross-lingual pretraining has significantly reduced the need for large parallel data.
Approach:	They compare unsupervised cross-lingual pretraining with masking and reconstructing inputs in the decoder to produce real sentences.
Outcome:	The proposed methods produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.

Copied to clipboard

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

CEPT: A Contrast-Enhanced Prompt-Tuning Framework for Emotion Recognition in Conversation (2024.lrec-main)

Challenge:	Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks.
Approach:	They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead.
Outcome:	The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks.

Copied to clipboard

Qingqing Gao, Jiuxin Cao, Biwei Cao, Xin Guan, Bo Liu

NSP-BERT: A Prompt-based Few-Shot Learner through an Original Pre-training Task —— Next Sentence Prediction (2022.coling-1)

Challenge:	Emotion recognition in conversation research suffers from data imbalance and the presence of similar linguistic expressions for different emotions.
Approach:	They propose a Contrast-Enhanced Prompt-Tuning framework that transforms an ERC task into a Masked Language Modeling task and generates the emotion for each utterance in the conversation.
Outcome:	The proposed framework outperforms the state-of-the-art methods on all three benchmark datasets and excels in recognizing minority emotions.

Copied to clipboard

Yi Sun, Yu Zheng, Chao Hao, Hangping Qiu

SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations (2021.findings-emnlp)

Challenge:	Recent studies have shown that using prompts to utilize language models to perform downstream tasks is more effective than using token-level methods such as PET.
Approach:	They propose to use a BERT original pre-training task abandoned by RoBERTa and other models to construct a sentence-level prompt-based method that does not need to fix the length of the prompt or the position to be predicted.
Outcome:	The proposed method performs better than PET and EFL on a BERT pre-training task and is comparable to other prompt-based methods.

Copied to clipboard

Hooman Sedghamiz, Shivam Raval, Enrico Santus, Tuka Alhanai, Mohammad Ghassemi

Parameter-free Automatically Prompting: A Latent Pseudo Label Mapping Model for Prompt-based Learning (2022.findings-emnlp)

Challenge:	SupCL-Seq extends contrastive learning from computer vision to sequence classification tasks.
Approach:	They propose a supervised alternative to Masked Language Modeling (MLM) that extends contrastive learning to sequence optimization in NLP by altering the dropout mask probability in standard Transformer architectures.
Outcome:	The proposed method leads to large gains on the GLUE benchmark, including 6% absolute improvement on CoLA, 5.4% on MRPC, 4.7% on RTE and 2.6% on STS-B.

Copied to clipboard

Jirui Qi, Richong Zhang, Junfan Chen, Jaein Kim, Yongyi Mao

Probing BERT’s priors with serial reproduction chains (2022.findings-acl)

Challenge:	Existing manual label mapping methods that require extra parameters and human knowledge are limited in data.
Approach:	They propose a Latent Pseudo Label Mapping method that optimizes the label mapping without human knowledge and extra parameters.
Outcome:	The proposed method outperforms the standard SOTA method in few-shot learning tasks and significantly outperformed the standard ALM method which requires extra task-specific prior knowledge.

Copied to clipboard

Takateru Yamakoshi, Thomas Griffiths, Robert Hawkins

Data-Efficient French Language Modeling with CamemBERTa (2023.findings-acl)

Challenge:	Large neural language models have induced surprisingly human-like linguistic knowledge, from syntactic structure and subtle lexical biases to more insidious social biase and stereotypes.
Approach:	They propose to use serial reproduction chains to generate representative samples from popular masked language models like BERT to test their hypothesis.
Outcome:	The proposed method is based on theories of iterated learning in cognitive science and can be used to probe masked language models.

Copied to clipboard

Wissam Antoun, Benoît Sagot, Djamé Seddah

Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022.acl-long)

Challenge:	Recent advances in NLP have significantly improved the performance of language models on a variety of tasks.
Approach:	They introduce a French DeBERTa model that builds upon the DeBERTAV3 architecture and training objective and evaluate its performance on a variety of French downstream tasks and datasets.
Outcome:	The proposed model outperforms BERT-based models on most tasks given the same amount of training tokens and trained on 30% of its input tokens.

Copied to clipboard

Yuan Chai, Yaobo Liang, Nan Duan

Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)

Challenge:	Multilingual pre-trained language models have shown impressive cross-lingual ability.
Approach:	They argue that cross-language ability comes from commonality between languages . they create an artificial language by modifying property in source language .
Outcome:	The proposed model can be implemented in multilingual and low-resource language scenarios without cross-lingual supervision or aligned data.

Copied to clipboard

Vijeta Deshpande, Dan Pechi, Shree Thatte, Vladislav Lialin, Anna Rumshisky

Self-Alignment Pretraining for Biomedical Entity Representations (2021.naacl-main)

Challenge:	Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered.
Approach:	They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language.
Outcome:	The proposed model performance is correlated with pre-training perplexity and performance.

Copied to clipboard

Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, Nigel Collier

Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models (2023.findings-acl)

Challenge:	Existing approaches to self-supervised learning of biomedical entities are limited in the biomedic domain.
Approach:	They propose a pretraining scheme that self-aligns the representation space of biomedical entities.
Outcome:	The proposed framework achieves state-of-the-art on six MEL benchmarking datasets.

Copied to clipboard

Somayeh Ghanbarzadeh, Yan Huang, Hamid Palangi, Radames Cruz Moreno, Hamed Khanpour

Context Analysis for Pre-trained Masked Language Models (2020.findings-emnlp)

Challenge:	Existing methods for debiasing are resource-intensive and costly. Existing solutions for debiansing require fine-tuning on downstream tasks.
Approach:	They propose to integrate Masked Language Modeling (MLM) training objectives into fine-tuning’s training process to debiase the PLMs.
Outcome:	The proposed approach outperforms the state-of-the-art baselines in terms of gender bias scores while improving PLMs’ performance solely using the downstream tasks’ dataset.

Copied to clipboard

Yi-An Lai, Garima Lalwani, Yi Zhang

Readability Controllable Biomedical Document Summarization (2022.findings-emnlp)

Challenge:	Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many downstream NLP tasks.
Approach:	They propose to use a masking and gradient approach to evaluate the impact of context on the word representation.
Outcome:	The proposed model architectures are architecture agnostic and gradient based.

Copied to clipboard

Zheheng Luo, Qianqian Xie, Sophia Ananiadou

DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition (2024.lrec-main)

Challenge:	Existing controllable summarization systems for biomedical documents have little attention to readability control, leaving users with incompatible summaries .
Approach:	They propose a task of readability controllable summarization for biomedical documents to generate summaries that are incompatible with users' levels of expertise.
Outcome:	The proposed model is based on pre-trained language models with prevalent controlling and generation techniques and evaluates the readability discrepancy between lay and technical summaries.

Copied to clipboard

Yi-Cheng Wang, Hsin-Wei Wang, Bi-Cheng Yan, Chi-Han Lin, Berlin Chen

Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020.acl-main)

Challenge:	End-to-end automatic speech recognition systems suffer from mistranscription of domain-specific phrases, such as named entities.
Approach:	They propose a named entity correction model that leverages phonetic con-fusion to mitigate phonetic confusion.
Outcome:	The proposed model outperforms the existing model on AISHELL-1 and Homophone datasets.

Copied to clipboard

Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui

Learning Better Masking for Better Language Model Pre-training (2023.acl-long)

Challenge:	Existing methods for incorporating a masked language model into an EncDec model have potential drawbacks when applied to GEC.
Approach:	They propose to incorporate a pre-trained masked language model (MLM) into an encoder-decoder model for grammatical error correction.
Outcome:	The proposed method achieves state-of-the-art on BEA-2019 and CoNLL-2014 benchmarks.

Copied to clipboard

Dongjie Yang, Zhuosheng Zhang, Hai Zhao

On Losses for Modern Language Models (2020.emnlp-main)

Challenge:	Existing PrLMs adopt a Random-Token Masking strategy with a fixed masking ratio and different contents are masked by an equal probability throughout the training.
Approach:	They propose two scheduled masking approaches that adaptively tune masking ratio and masked content in different training stages, which improves pre-training efficiency and effectiveness.
Outcome:	The proposed methods improve the pre-training efficiency and effectiveness on the downstream tasks.

Copied to clipboard

Stéphane Aroca-Ouellette, Frank Rudzicz

Probing Simile Knowledge from Pre-trained Language Models (2022.acl-long)

Challenge:	Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP).
Approach:	They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train.
Outcome:	The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens.

Copied to clipboard

Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, Chang Su

On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies (2021.naacl-main)

Challenge:	Existing approaches to learn generic knowledge from a large corpus are time-consuming and labor-intensive.
Approach:	They propose a framework to probe simile knowledge from pre-trained language models to solve SI and SG tasks.
Outcome:	The proposed framework solves the SI and SG tasks in a simile triple completion task.

Copied to clipboard

Tianyi Zhang, Tatsunori B. Hashimoto

Effective Pretraining Objectives for Transformer-based Autoencoders (2022.findings-emnlp)

Challenge:	Recent studies suggest that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks.
Approach:	They construct cloze-like masks using task-specific lexicons to explain their results . they show that the majority of performance gains come from generic masks that are not associated with the lexical .
Outcome:	The proposed method outperforms a classic method for unsupervised parsing.

Copied to clipboard

Luca Di Liello, Matteo Gabburo, Alessandro Moschitti

Contrastive Learning for Prompt-based Few-shot Language Learners (2022.naacl-main)

Challenge:	ELECTRA is more accurate than BERT, but it is not clear if this is due to its innovative architecture or to the long and extensive training, which highly increases the computation cost for obtaining the final language model.
Approach:	They propose to replace BERT’s Masked Language Modeling objective (MLM) with Token Detection (TD) by using a statistical approach to generate light tokens.
Outcome:	The proposed method can replace ELECTRA's computationally heavy generators without a significant drop in performance.

Copied to clipboard

Yiren Jian, Chongyang Gao, Soroush Vosoughi

SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)

Challenge:	a recent study has shown that GPT-3 fine-tuning models with limited examples is effective . a contrastive learning framework clusters inputs from the same class under different augmented “views” and repels those from different classes.
Approach:	They propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" they combine a contrastive loss with the standard masked language modeling loss in prompt-based few-shot learners .
Outcome:	The proposed framework improves on the state-of-the-art methods in a diverse set of 15 language tasks.

Copied to clipboard

Erik Nijkamp, Bo Pang, Ying Nian Wu, Caiming Xiong

Detecting Sexual Content at the Sentence Level in First Millennium Latin Texts (2024.lrec-main)

Challenge:	Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task.
Approach:	They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines .
Outcome:	The proposed model improves sample-efficiency and performance over strong baselines.

Copied to clipboard

Thibault Clerice

Information Retrieval

Challenge:	a traditional approach to corpus building involves constructing a corpus centered around specific themes, such as colors.
Approach:	They propose to use deep learning methods to accelerate corpus building in humanities . they propose to integrate metadata embeddings into the model to improve accuracy .
Outcome:	The proposed method outperforms token-based searches in the humanities and linguistics field.

Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure (2024.naacl-long)

Copied to clipboard

David Arps, Laura Kallmeyer, Younes Samih, Hassan Sajjad

Syntactic Text Processing

Challenge:	a number of studies have focused on making explicit the linguistic information encoded in language models (LMs) however, this method has been criticized for various reasons.
Approach:	They introduce a framework for creating nonce treebanks for multilingual UD corpora . they investigate word co-occurrence statistics and show how nonce data affects the performance of syntactic dependency probes.
Outcome:	The proposed framework satisfies syntactic argument structure and ensures grammaticality via language-specific rules.

Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)

Copied to clipboard

Xiaotao Gu, Yikang Shen, Jiaming Shen, Jingbo Shang, Jiawei Han

Low-Resource NLP

Challenge:	Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task.
Approach:	They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures.
Outcome:	The proposed method improves the identification of high-level structures using phrase-guided masking.

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (D19-1)

Copied to clipboard

Elena Voita, Rico Sennrich, Ivan Titov

Comparing Text Representations: A Theory-Driven Approach (2021.emnlp-main)

Challenge:	a recent study has shown that deep neural networks are effective with various tasks . a new study examines how representations of tokens evolve between layers under different learning objectives .
Approach:	They use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers.
Outcome:	The proposed model outperforms untrained models on word identity prediction tasks . the model outpersforms models trained on other linguistic tasks based on the model's objective .

Copied to clipboard

Gregory Yauney, David Mimno

Text Classification

Challenge:	Recent advances in NLP have been made by learning representations that transform complex tasks into simple classification tasks.
Approach:	They propose a method to evaluate the compatibility between representations and tasks by fitting text features to specific characteristics of text datasets.
Outcome:	The proposed model provides a calibrated, quantitative measure of the difficulty of a classification-based NLP task.

Unsupervised Sentence Representation Learning with Syntactically Aligned Negative Samples (2025.findings-naacl)

Copied to clipboard

Zhilan Wang, Zekai Zhi, Rize Jin, Kehui Song, He Wang, Da-Jung Cho

XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation (2022.emnlp-main)

Challenge:	Existing approaches to sentence representation learning often encounter semantic inconsistencies and feature suppression.
Approach:	They propose a method for generating syntactically aligned negative (SAN) samples using a semantic importance-aware Masked Language Model (MLM) approach.
Outcome:	The proposed method produces negative samples with substantial textual overlap with the original sentences while conveying different meanings.

Copied to clipboard

Yong Wang, Shilin He, Guanhua Chen, Yun Chen, Daxin Jiang

Evaluation of HTR models without Ground Truth Material (2022.lrec-1)

Challenge:	Existing pre-training language models have been successful in natural language understanding and autoregressive generation tasks, but non-autoregressive models have not been sufficiently successful.
Approach:	They propose a pre-trained masked language model (MLM) and a non-autoregressive generation model with a lightweight decorator.
Outcome:	The proposed model outperforms the previous mask-predict model on translation datasets by 19.9x.

Copied to clipboard

Phillip Benjamin Ströbel, Martin Volk, Simon Clematide, Raphael Schwitter, Tobias Hodel, David Schoch

Multilingual Pre-training with Self-supervision from Global Co-occurrence Information (2023.findings-acl)

Challenge:	Optical Character Recognition (OCR) is a well-established technique for digitising historical printed collections in libraries and archives.
Approach:	They propose to use masked language models to evaluate handwritten text recognition models . they propose to introduce GT-free metrics to evaluate models to ensure best results .
Outcome:	The proposed model evaluations are based on lexicon-based and masked language models.

Copied to clipboard

Xi Ai, Bin Fang

Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)

Challenge:	Empirical studies show multilinguality and crosslinguality emerge from MLM pretraining without supervision.
Approach:	They propose to use global co-occurrence information as a source of structural information on multilingual corpora.
Outcome:	Empirical studies show that MLM-GC pre-training outperforms MLM pre- training for 4 downstream cross-lingual tasks and 1 additional monolingual task.

Copied to clipboard

Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, Aitor Soroa

Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models (2020.emnlp-main)

Challenge:	Large language models require huge training corpora, which is unobtainable for most NLP practitioners.
Approach:	They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget.
Outcome:	The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics.

Copied to clipboard

Thuy-Trang Vu, Dinh Phung, Gholamreza Haffari

Universal Sentence Representation Learning with Conditional Masked Language Model (2021.emnlp-main)

Challenge:	Recent work has shown the importance of training contextualised word embedding models on the domain of the target task of interest.
Approach:	They propose a masking strategy which adversarially masks out those tokens which are harder to reconstruct by the underlying MLM.
Outcome:	The proposed training strategy outperforms random masking on six unsupervised domain adaptation tasks and achieves up to +1.64 F1 score improvements.

Copied to clipboard

Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, Eric Darve

Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking (2022.emnlp-main)

Challenge:	Existing methods to learn sentence representations on unlabeled corpora are difficult and expensive to obtain, making it hard to cover many domains and languages.
Approach:	They propose a method to train sentence representations on large unlabeled corpora by conditioning on the encoded vectors of adjacent sentences.
Outcome:	The proposed method outperforms existing models on SentEval and can be extended to a broad range of languages and domains.

Copied to clipboard

Mingyu Lee, Jun-Hyung Park, Junho Kim, Kang-Min Kim, SangKeun Lee

Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource Languages (2022.emnlp-main)

Challenge:	Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations but comes at a substantial training cost.
Approach:	They propose a concept-based curriculum masking method that evaluates the MLM difficulty of each token based on a carefully-designed linguistic difficulty criterion.
Outcome:	The proposed method significantly improves pre-training efficiency with the original BERT model at half the training cost.

Copied to clipboard

Olga Pelloni, Anastassia Shaitarova, Tanja Samardzic

Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation (2023.acl-long)

Challenge:	English is the most natural choice for cross-lingual transfer, but it is often not the best choice for low-resource languages.
Approach:	They propose to use pre-trained multilingual models to improve performance in low-resource languages via cross-lingual transfer.
Outcome:	The results show that languages written in non-Latin and non-alphabetic scripts are the best choices for improving performance on Masked Language Modelling tasks in a diverse set of 30 low-resource languages.

Copied to clipboard

Xiaohang Tang, Yi Zhou, Danushka Bollegala

Swap and Predict – Predicting the Semantic Changes in Words across Corpora by Context Swapping (2023.findings-emnlp)

Challenge:	Existing methods for learning dynamic contextualised word embeddings do not capture temporal semantic variations of words.
Approach:	They propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model using time-sensitive templates.
Outcome:	The proposed method significantly reduces the perplexity of test sentences in C2 outperforming the current state-of-the-art.

Copied to clipboard

Taichi Aida, Danushka Bollegala

Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks (2022.emnlp-main)

Challenge:	Detecting semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions.
Approach:	They propose a method that randomly swaps contexts between two different corpora to detect whether a given word changes its meaning . they then use a pretrained masked language model to generate contextualised word embeddings of w, which are then used to predict the semantic changes of words in four languages .
Outcome:	The proposed method achieves significant performance improvements compared to baselines for the English semantic change prediction task.

Copied to clipboard

Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, Reza Shokri

On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)

Challenge:	Prior attempts at measuring leakage of MLMs via membership inference attacks have been inconclusive, implying potential robustness of Mlms to privacy attacks.
Approach:	They propose a stronger membership inference attack based on likelihood ratio hypothesis testing that involves an additional MLM to more accurately quantify the privacy risks of memorization in MLMs.
Outcome:	The proposed attack improves the AUC of prior membership inference attacks from 0.66 to an alarmingly high 0.90 level on models trained on medical notes.

Copied to clipboard

Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa

Unsupervised Data Augmentation for Aspect Based Sentiment Analysis (2022.coling-1)

Challenge:	Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models.
Approach:	They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning.
Outcome:	The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task.

Copied to clipboard

David Z. Chen, Adam Faulkner, Sahil Badyal

Information Extraction & Text Mining

Challenge:	Recent approaches to Aspect-based Sentiment Analysis (ABSA) perform the subtasks of aspect term extraction (ATE) and aspect sentiment classification (ASC) simultaneously.
Approach:	They introduce an adaptation of Unsupervised Data Augmentation in semi-supervised learning that performs both aspects of Aspect-based Sentiment Analysis (ABSA) and aspect sentiment classification (ASC) they show that simple augmentations applied to modest-sized datasets along with consistency training lead to competitive performance with current ABSA state-of-the-art in restaurant and laptop domains .
Outcome:	The proposed approach performs well on a span-level classification task with minimal training data.

Event Representation Learning with Multi-Grained Contrastive Learning and Triple-Mixture of Experts (2024.lrec-main)

Copied to clipboard

Tianqi Hu, Lishuang Li, Xueyang Qin, Yubo Feng

Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models (2024.findings-emnlp)

Challenge:	Existing methods for learning event representations fail to capture hidden feature information between events.
Approach:	They propose a method that extends the random masked language model by incorporating a specialized MLM to capture different grammatical structures within events.
Outcome:	The proposed method outperforms baselines in hard similarity and transitive sentence similarity tasks, highlighting the superiority of the proposed method.

Copied to clipboard

Shitian Zhao, Renrui Zhang, Xu Luo, Yan Wang, Shanghang Zhang, Peng Gao

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models (2023.findings-acl)

Challenge:	Existing multi-modal language models with different architectures, parameter sizes, training datasets, and pipelines exhibit varying strengths across different tasks.
Approach:	They propose a framework for fusing heterogeneous models off-the-shell, which they call likelihood composition, and introduce basic operations to compose multiple models’ likelihood distribution when doing a multi-choice visual-question-answering task.
Outcome:	The proposed framework can be used to fusing heterogeneous models off-the-shell.

Copied to clipboard

Saleh Soltan, Andy Rosenbaum, Tobias Falke, Qin Lu, Anna Rumshisky, Wael Hamza

A Two-Stage Masked LM Method for Term Set Expansion (2020.acl-main)

Challenge:	Pre-trained encoder-only and sequence-to-sequence models are computationally expensive.
Approach:	They propose a recipe to initialize one model from the other to improve pre-training efficiency.
Outcome:	The proposed method matches the performance of a from-scratch model with a multilingual encoder while reducing the total compute cost by 27%.

Copied to clipboard

Guy Kushilevitz, Shaul Markovitch, Yoav Goldberg

How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)

Challenge:	Existing methods for Term Set Expansion are either distributional or pattern-based . Term set expansion is a task of expanding a small seed set of example terms into a larger set of terms that belong to the same semantic category.
Approach:	They propose a method which uses neural masked language models to expand a small seed set of terms into a larger set of semantic terms.
Outcome:	The proposed method outperforms state-of-the-art methods due to the small seed set size . it uses neural masked language models to query large, pre-trained mlms .

Copied to clipboard

Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita, Gaku Morio, Yasuhiro Sogawa

Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking (2023.findings-acl)

Challenge:	Masked language modeling (MLM) is a widely used self-supervised pretraining objective.
Approach:	They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context.
Outcome:	The proposed objectives show that they should have half the complexity needed to perform comparably to MLM.

Copied to clipboard

Inigo Urteaga, Moulay Zaidane Draidia, Tomer Lancewicki, Shahram Khadivi

Masked Audio Text Encoders are Effective Multi-Modal Rescorers (2023.findings-acl)

Challenge:	Using a Bayesian optimization framework, we pre-train Transformer-based language models (TLMs) using a multi-armed bandit framework requires high computational resources and introduces many unresolved design choices.
Approach:	They propose a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models.
Outcome:	The proposed framework achieves lower MLM loss in fewer epochs, across settings, while avoiding expensive hyperparameter grid search.

Copied to clipboard

Jinglun Cai, Monica Sunkara, Xilai Li, Anshu Bhatia, Xiao Pan, Sravan Bodapati

A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models (2023.emnlp-main)

Challenge:	Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition systems.
Approach:	They propose a multi-modal masked language model rescorer which integrates acoustic representations into the input space of MLM.
Outcome:	The proposed model reduces word error rate (WER) by 4%-16% on in-domain and 3%-7% on out-of-domain datasets over the text-only baseline.

Copied to clipboard

Yi Zhou, Jose Camacho-Collados, Danushka Bollegala

Contextualized Semantic Distance between Highly Overlapped Texts (2023.findings-acl)

Challenge:	Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work.
Approach:	They conduct a comprehensive study on 39 pretrained MLMs to examine their model factors and their social biases.
Outcome:	The proposed model factors influence social biases learned by an MLM and their downstream task performance.

Copied to clipboard

Letian Peng, Zuchao Li, Hai Zhao

Unsupervised Text Style Transfer with Padded Masked Language Models (2020.emnlp-main)

Challenge:	Conventional semantic metrics are based on word representations and are vulnerable to disturbance of overlapped components with similar representations.
Approach:	They propose a mask-and-predict strategy to evaluate the semantic distance between the overlapped sentences using words in the longest common sequence as neighboring words and use masked language modeling to predict their positions.
Outcome:	The proposed method outperforms the state-of-the-art in domain adaption by a huge margin.

Copied to clipboard

Eric Malmi, Aliaksei Severyn, Sascha Rothe

Low-Resource NLP

Challenge:	Existing methods for style transfer are difficult to obtain and require substantial amounts of parallel training examples to work well.
Approach:	They propose an unsupervised method for style transfer that uses masked language models to find the text spans where the two models disagree the most in terms of likelihood.
Outcome:	The proposed method performs competitively in a fully unsupervised setting and improves accuracy in low-resource settings by over 10 percentage points when pre-training on silver training data generated by Masker.

NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)

Copied to clipboard

Nikolay Arefyev, Dmitrii Kharchev, Artem Shelmanov

Debias NLU Datasets via Training-free Perturbations (2023.findings-emnlp)

Challenge:	Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding.
Approach:	They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.
Outcome:	The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.

Copied to clipboard

Qi Guo, Yuanhang Tang, Yawen Ouyang, Zhen Wu, Xinyu Dai

Robustness in NLP

Challenge:	Existing approaches to debiase NLU models capture biased features that are independent of the task but spuriously correlated to labels.
Approach:	They propose a framework that conducts training-free perturbations on samples containing biased features to Debias NLU Datasets.
Outcome:	The proposed framework shows competitive performance with previous state-of-the-art debiasing strategies.

AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese (2023.findings-acl)

Copied to clipboard

Abhijnan Nath, Sheikh Mannan, Nikhil Krishnaswamy

CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)

Challenge:	AxomiyaBERTa is a novel BERT model for low-resource languages . Transformers require extensive computing resources and suffer in low-compute settings .
Approach:	They propose a novel BERT model for Assamese, a morphologically-rich low-resource language of eastern India that is trained on a simple masked language modeling task without the NSP objective.
Outcome:	The proposed model performs well on token-level tasks and on “longer context” tasks with the aid of embedding disperser and phonological signals.

Copied to clipboard

Jingwei Zhang, Xin Wu, Yi Cai

DMLM: Descriptive Masked Language Modeling (2023.findings-acl)

Challenge:	Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension.
Approach:	They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images .
Outcome:	The proposed method improves performance 37.94% on referring expression comprehension task.

Copied to clipboard

Edoardo Barba, Niccolò Campolungo, Roberto Navigli

Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)

Challenge:	Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition.
Approach:	They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition.
Outcome:	The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling.

Copied to clipboard

David Dukić, Jan Snajder

On the Difference of BERT-style and CLIP-style Text Encoders (2023.findings-acl)

Challenge:	Pre-trained language models excel in natural language understanding (NLU) tasks.
Approach:	They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance.
Outcome:	The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear.

Copied to clipboard

Zhihong Chen, Guiming Chen, Shizhe Diao, Xiang Wan, Benyou Wang

Visual Data in NLP

Challenge:	Masked language modeling is one of the most popular pretraining recipes in natural language processing.
Approach:	They analyze BERT-style and CLIP-style text encoders from three experiments . they show that CLIP style encoder is equipped with synesthesia for the cross-modal association .
Outcome:	The proposed models outperform BERT-style models on vision-centric text understanding tasks, but have synesthesia for the cross-modal association, similar to the senses of humans.

Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)

Copied to clipboard

Gábor Berend

Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification (2023.findings-emnlp)

Challenge:	a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance .
Approach:	They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible .
Outcome:	The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes .

Copied to clipboard

Yookyung Kho, Jaehee Kim, Pilsung Kang

ORTicket: Let One Robust BERT Ticket Transfer across Different Tasks (2024.lrec-main)

Challenge:	Recent prompt-based fine-tuning techniques have garnered considerable interest as a core technique for few-shot text classification tasks.
Approach:	They propose a prompt-based fine-tuning approach that reformulates the fine-uning objective to align with the Masked Language Modeling objective.
Outcome:	The proposed method has shown superior performance on five multi-class classification datasets.

Copied to clipboard

Yuhao Zhou, Wenxiang Chen, Rui Zheng, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang

CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents (2025.findings-acl)

Challenge:	Pretrained language models are susceptible to subtle perturbations and require multiple adversarial training during fine-tuning to improve their robustness.
Approach:	They propose a novel adversarial defense method ORTicket that fine-tunes a model for downstream tasks.
Outcome:	The proposed method achieves comparable robustness to other defense methods while maintaining the efficiency of fine-tuning.

Copied to clipboard

Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, Guohao Li

Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them (2025.findings-emnlp)

Challenge:	Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexity of constructing tasks and evaluators.
Approach:	They propose a cross-environment agent benchmark framework that integrates graph-based evaluation and task generation methods.
Outcome:	The proposed framework supports multiple devices and can be easily extended to any environment with a Python interface.

Copied to clipboard

Marc Felix Brinner, Tarek Al Mustafa, Sina Zarrieß

Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection (2025.findings-acl)

Challenge:	a new method for continual pretraining transformer encoder models is proposed for specialized domains with limited training data.
Approach:	They propose to use LLM-generated data to enrich domain-specific ontologies and pretrain transformer encoder models as an ontology-informed embedding model for concept definitions.
Outcome:	The proposed method improves on standard MLM pretraining on invasion biology domains.

Copied to clipboard

San Kim, Jonghwi Kim, Yejin Jeon, Gary Lee

Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation (2026.findings-acl)

Challenge:	Retrieval-Augmented Generation (RAG) provides external knowledge for accurate and up-to-date responses, but external knowledge is vulnerable to poisoning and unauthorized injections.
Approach:	They propose a Gradient-based Masked Token Probability defense method to detect and filter out adversarially crafted documents by examining gradients of the retriever’s similarity function.
Outcome:	Experiments show that the proposed method eliminates over 90% of poisoned content while retaining relevant documents.

Copied to clipboard

Seok Hwan Song, Azher Ahmed Efat, Wallapak Tavanapong

The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology (2025.emnlp-main)

Challenge:	Existing studies have not examined the y-axis-related biases in chart-to-table translation models.
Approach:	They propose a framework for analyzing y-axis-related bias on five state-of-the-art models.
Outcome:	The proposed framework analyzes y-axis-related biases on five state-of-the-art models.

Copied to clipboard

Fagun Patel, Duc Quang Nguyen, Sang T. Truong, Jody Vaynshtok, Sanmi Koyejo, Nick Haber

Training compute-optimal transformer encoder models (2025.emnlp-main)

Challenge:	State-of-the-art multimodal language models (MLMs) show promise for supporting SLPs, but their use remains underexplored due to a limited understanding of their performance in high-stakes clinical settings.
Approach:	They propose a taxonomy of real-world use cases of multimodal language models in speech-language pathologies to address this gap.
Outcome:	The proposed model outperforms 15 state-of-the-art models in speech-language pathologies across five use cases and achieves improvements of over 30% on domain-specific data.

Copied to clipboard

Megi Dervishi, Alexandre Allauzen, Gabriel Synnaeve, Yann LeCun