Papers with pretraining
MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (2026.acl-industry)
Copied to clipboard
Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Zirui Tang, Boyu Niu, Yuanhong Zheng, Dongsheng Ma, Ziyang Miao, Hejun Dong, Siyi Qian, Junyuan Zhang, Fangdong Wang, Jingzhou Chen, Xiaomeng Zhao, Liqun Wei, Wei Li, Shasha Wang, RuiLiang Xu, Yuanyuan Cao, Lu Chen, Qianqian Wu, Huaiyu Gu, Lindong Lu, Dechen Lin, null Shenguanlin, Xuanhe Zhou, Linfeng Zhang, Yuhang Zang, Xiaoyi Dong, Jiaqi Wang, Bo Zhang, Lei Bai, Pei Chu, Weijia Li, Jiang Wu, Lijun Wu, Zhenxiang Li, Guangyu Wang, Zhongying Tu, Chao Xu, Kai Chen, Bowen Zhou, Dahua Lin, Wentao Zhang, Conghui He
| Challenge: | Document images are characterized by higher resolutions, denser content, and more complex structural layouts. |
| Approach: | They propose a 1.2B-parameter document parsing vision-language model that decouples layout analysis from local content recognition. |
| Outcome: | The proposed model surpasses general-purpose and domain-specific models on multiple benchmarks while maintaining significantly lower computational overhead. |
Topic Modeling for Short Texts with Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be used to solve topic modeling challenges for short texts by contextually learning the meanings of words. |
| Approach: | They propose two approaches to using Large Language Models (LLMs) for topic modeling: parallel prompting and sequential prompting. |
| Outcome: | The proposed methods identify more coherent topics than existing ones while maintaining the diversity of the induced topics. |
Towards Unifying the Label Space for Aspect- and Sentence-based Sentiment Analysis (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train ABSA model are limited by lack of annotated data . a dual-granularity pseudo labeling approach is proposed to solve this problem . |
| Approach: | They propose a framework for aspect-based sentiment analysis that uses annotated data to train ABSA models. |
| Outcome: | The proposed framework surpasses previous methods on benchmarks. |
GAP-Gen: Guided Automatic Python Code Generation (2023.eacl-srw)
Copied to clipboard
| Challenge: | Several previous approaches convert a sentence into a formal statement by mapping verbs to functions in the formal language. |
| Approach: | They propose a Guided Automatic Python Code Generation method based on Python syntactic constraints and semantic constraints. |
| Outcome: | The proposed method achieves better results on automatic Python code generation task than previous methods. |
Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models (2025.tacl-1)
Copied to clipboard
| Challenge: | a recent study revisits six core challenges that have influenced the evolution of Neural Machine Translation (NMT) domain mismatch, amount of parallel data, rare word prediction, translation of long sentences and sub-optimal beam search remain challenges in LLMs. |
| Approach: | They revisit core challenges that have acted as benchmarks for progress in NMT . they propose to revisit these challenges and offer insights into their relevance . |
| Outcome: | The proposed models significantly improve translation of sentences containing approximately 80 words, even translating documents up to 512 words. |
Rethinking Full Finetuning from Pretraining Checkpoints in Active Learning for African Languages (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches to improve model performance are finetuning on all acquired data after each round, which is computationally expensive in multilingual and low-resource settings. |
| Approach: | They evaluate continual finetuning (CF) against full finetuned (FA) across 28 African languages using MasakhaNEWS and SIB-200. |
| Outcome: | The proposed approach outperforms full finetuning (FA) in 28 African languages, achieving up to 35% reductions in GPU memory, FLOPs, and training time. |
Answer-Supervised Question Reformulation for Enhancing Conversational Machine Comprehension (D19-58)
Copied to clipboard
| Challenge: | Existing question reformulation models are based on supervised question labels without considering feedback information from answers. |
| Approach: | They propose a question reformulation model that integrates conversational history information with reinforcement learning. |
| Outcome: | The proposed model is more effective in conversational machine comprehension with reinforcement learning. |
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |
Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques . |
| Approach: | They aim to introduce techniques for learning from little-to-no data using pretrained language models. |
| Outcome: | This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods . |
Book QA: Stories of Challenges and Opportunities (D19-58)
Copied to clipboard
| Challenge: | Existing approaches to answer questions based on the full text of books are limited by their unique characteristics. |
| Approach: | They propose a system for answering questions based on the full text of books . they use a memory network to reason and predict an answer, and a novel question generator to improve generalization. |
| Outcome: | The proposed system improves on the recently published NarrativeQA corpus on Who questions . it shows that the proposed system is highly challenging and needs more research . |
Multi-Document Summarization with Centroid-Based Pretraining (2023.acl-short)
Copied to clipboard
| Challenge: | In Multi-Document Summarization, the input is a set of documents, and the output is its summary. |
| Approach: | They propose a novel pretraining objective that uses the ROUGE-based centroid of each document cluster as a proxy for its summary. |
| Outcome: | The proposed model is better or comparable to state-of-the-art models. |
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages (2021.eacl-srw)
Copied to clipboard
| Challenge: | Neural dependency parsing has been a success for many domains and languages, but the bottleneck of massive labelled data limits its effectiveness for low resource languages. |
| Approach: | They propose to use morphological knowledge to improve dependency parsing for morphology rich languages in a low-resource setting to perform experiments. |
| Outcome: | The proposed method achieves an average gain of 2 points (UAS) and 3.6 points (LAS) on 10 MRLs in low-resource settings. |
Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon (2024.eacl-long)
Copied to clipboard
| Challenge: | Prior work extended multilingual models to other languages due to the unavailability of labeled and unlabeled training data. |
| Approach: | They use multilingual lexicons to enhance multilingual models capabilities in low-resource languages . they focus on zero-shot sentiment analysis tasks across 34 languages based on a single sentence . |
| Outcome: | The proposed model improves zero-shot performance across 34 languages without using any sentence-level sentiment data. |
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining. |
| Approach: | They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora. |
| Outcome: | The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages. |
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. |
| Approach: | They propose a method that explicitly models VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve extraction accuracy. |
| Outcome: | The proposed methods show that inserting special tokens denoting layout group boundaries can lead to a 1.9% Macro F1 improvement in token classification. |
Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing theoretical frameworks for large language models (LLMs) do not explain how pretraining leads to in-context learning. |
| Approach: | They propose a theoretical framework that allows LLMs to generalize to unseen instructions and perform in-context learning even when verbalizers are irrelevant to the task. |
| Outcome: | The proposed framework can be used to analyze LLMs' ability to perform in-context learning . it can be applied to linguistic, psychology, and philosophy tasks . |
Towards efficient self-supervised representation learning in speech processing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models require several GPUs for days to pretrain, generating environmental concerns because of their high energy consumption. |
| Approach: | They propose an efficient self-supervised model that uses a single GPU during 24 to 48 hours of pretraining to address high computational costs. |
| Outcome: | The proposed model represents two orders of magnitude better than existing models. |
More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline. |
| Approach: | They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects. |
| Outcome: | The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context. |
Cross-domain Analysis on Japanese Legal Pretrained Language Models (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing studies do not care the performance of domain-adapted PLMs for a generic domain. |
| Approach: | They propose to use pretraining strategies to build pretrained language models specialised in the legal domain to improve their performance. |
| Outcome: | The pretrained language models can learn domain-specific and general word meanings simultaneously and can distinguish them. |
Overcoming Poor Word Embeddings with Word Definitions (2021.starsem-1)
Copied to clipboard
| Challenge: | Modern natural language understanding models depend on pretrained word embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. |
| Approach: | They propose a method to improve a model's ability to learn to use definitions in natural text to overcome this handicap. |
| Outcome: | The proposed model learns to use definitions in natural text to overcome this handicap. |
D-NET: A Pre-Training and Fine-Tuning Framework for Improving the Generalization of Machine Reading Comprehension (D19-58)
Copied to clipboard
Hongyu Li, Xiyuan Zhang, Yibing Liu, Yiming Zhang, Quan Wang, Xiangyang Zhou, Jing Liu, Hua Wu, Haifeng Wang
| Challenge: | MRC requires machines to understand text and answer questions about the text. |
| Approach: | They propose a simple system Baidu submitted for MRQA 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. |
| Outcome: | The proposed system is ranked at top 1 of all participants in terms of averaged F1 score. |
Few-shot fine-tuning SOTA summarization models for medical dialogues (2022.naacl-srw)
Copied to clipboard
| Challenge: | Abstractive summarization of medical dialogues is a challenge for standard training approaches due to the paucity of suitable datasets. |
| Approach: | They propose to use medical dialogues to generate abstractive summaries using transformer-based models with zero-shot and few-shot learning strategies. |
| Outcome: | The proposed models were compared with a medical dialogue dataset with 143 snippets and a general domain and dialogue-specific text to assess their performance. |
Do LLMs Really Know What They Don’t Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work suggests that large language models (LLMs) produce hallucinated and factually correct outputs. |
| Approach: | They propose a taxonomy categorizing hallucinations into Unassociated Hallucination (UH) and Associated Hallucinian (AH) they propose to use internal signals to distinguish hallucinos from factual errors . |
| Outcome: | The proposed taxonomy categorizes hallucinations into Unassociated Hallucination (UH) and Associated Hallucinications (AHs) based on the proposed taxonomic, the authors show that hidden states reflect whether the model is recalling parametric knowledge rather than the truthfulness of the output itself. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
Transferring General Multimodal Pretrained Models to Text Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text recognition rely on large-scale pretraining on human-annotated or synthetic data. |
| Approach: | They propose a method to transfer multimodal pretrained models to text recognition using image captioning. |
| Outcome: | The proposed method outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. |
Sign Language Translation with Sentence Embedding Supervision (2024.acl-short)
Copied to clipboard
| Challenge: | State-of-the-art sign language translation systems facilitate learning through gloss annotations when available at scale. |
| Approach: | They propose to use sentence embeddings of the target sentences at training time that take the role of glosses to supervise the learning process. |
| Outcome: | The proposed method significantly outperforms gloss-free approaches on German and American sign languages and with mono- and multilingual sentence embeddings and translation systems. |
A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | supervised word alignment tools such as GIZA++, MGIZA (Gao and Vogel, 2008) and FastAlign remain stagnant in terms of word alignment accuracy. |
| Approach: | They propose a supervised word alignment method based on cross-language span prediction by formalizing a word alignment problem as a collection of independent predictions from a token in the source sentence to a span in the target sentence. |
| Outcome: | The proposed method significantly outperforms previous supervised and unsupervised word alignment methods without any bitexts for pretraining. |
Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)
Copied to clipboard
| Challenge: | Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate. |
| Approach: | They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively. |
| Outcome: | The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines. |
Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing probing datasets focus on knowledge about relations between words and entities. |
| Approach: | They propose a new probing task to evaluate word understanding directly using dictionary definitions of words. |
| Outcome: | The proposed task evaluates word understanding directly using dictionary definitions of words. |
Evaluating Pretraining Strategies for Clinical BERT Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing generic language models in specialized domains may be sub-optimal due to domain differences. |
| Approach: | They propose various strategies for adapting a generic language model to the target domain and various forms of vocabulary modifications to fine-tune it. |
| Outcome: | The proposed strategies outperform a general-domain language model but little difference in performance between the models. |
Personal Information Parroting in Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models memorize millions of PI instances, increasing privacy risks. |
| Approach: | They develop a model that parrots 13.6% of PI verbatim on a manually curated set of 483 instances . they recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting. |
| Outcome: | The proposed model outperforms the best regex-based PI detectors on a manually curated set of 483 instances of PI. |
A Family of Pretrained Transformer Language Models for Russian (2024.lrec-main)
Copied to clipboard
Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, Alena Fenogenova
| Challenge: | Developing Transformer language models for the Russian language has received little attention . most of these LMs are developed for English, which imposes substantial constraints on the potential of the language technologies. |
| Approach: | They propose to release 13 Russian Transformer language models that span three languages . they aim to broaden the scope of NLP research directions and develop industrial solutions for the Russian language. |
| Outcome: | The proposed models are based on Russian language datasets and benchmarks. |
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability (2025.emnlp-demos)
Copied to clipboard
Mohammad Aflah Khan, Ameya Godbole, Johnny Wei, Ryan Yixiang Wang, James Flemings, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
| Challenge: | Existing workflows for pretraining large language models are cumbersome, fragmented and inaccessible. |
| Approach: | They propose an open-source library for editing, inspection, and analysis of large language model datasets. |
| Outcome: | TokenSmith is an open-source library for editing, inspection, and analysis of large language model datasets. |
On Relation-Specific Neurons in Large Language Models (2025.emnlp-main)
Copied to clipboard
Yihong Liu, Runsheng Chen, Lea Hirlimann, Ahmad Dawar Hakimi, Mingyang Wang, Amir Hossein Kargaran, Sascha Rothe, François Yvon, Hinrich Schuetze
| Challenge: | In large language models, certain neurons can store distinct pieces of knowledge learned during pretraining. |
| Approach: | They hypothesize that relation-specific neurons detect relation in input text and guide generation involving such a relation. |
| Outcome: | The proposed model can handle facts involving relation r and facts containing a different relation . |
Time Sensitive Knowledge Editing through Efficient Finetuning (2024.acl-short)
Copied to clipboard
Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, Yunyao Li
| Challenge: | Existing locate-and-edit knowledge editing methods suffer from two limitations: they are infeasible for large scale KE in practice and require long run-time. |
| Approach: | They propose to use parametric fine-tuning techniques to update obsolete knowledge and induce new knowledge into LLMs. |
| Outcome: | The proposed methods improve the performance of KE and knowledge update in a temporal dataset with knowledge update and knowledge injection examples. |
Probing Critical Learning Dynamics of PLMs for Hate Speech Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on pretrained language models (PLMs) for hate speech detection have not investigated how their performance is affected by pretraining and finetuning. |
| Approach: | They propose to compare pretrained language models, evaluate their seed robustness, finetuning settings, and the impact of pretraining data collection time. |
| Outcome: | The proposed models show that they are more robust than other models and that they have a better chance of performing better than domain-specific models. |
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs (2025.emnlp-main)
Copied to clipboard
Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao
| Challenge: | Multimodal Large Language Models (MLLMs) integrate visual and textual inputs, yet modality alignment remains one of the most challenging aspects. |
| Approach: | They propose a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. |
| Outcome: | The proposed method improves performance across various model sizes, with smaller models benefiting the most. |
PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for multilingual sequence-to-sequence pretraining rely on monolingual corpora and do not use strong cross-lingual signal contained in parallel data. |
| Approach: | They propose a method that replaces monolingual words with a bilingual dictionary and predicts the reference translation according to a parallel corpus instead of recovering the original sequence. |
| Outcome: | The proposed method improves machine translation and cross-lingual natural language inference by 2.0 BLEU points and 6.7 accuracy points over existing methods at a fraction of their computational cost. |
Permutation Invariant Strategy Using Transformer Encoders for Table Understanding (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for encoding text in tables require additional training and require additional pretraining. |
| Approach: | They propose a novel encoding strategy that preserves the critical property of permutation invariance across rows or columns. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on three table interpretation tasks: column type annotation, relation extraction, and entity linking. |
Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition (2020.acl-main)
Copied to clipboard
| Challenge: | Many studies have applied reinforcement learning to train a dialog policy . but modeling a real-world user simulator is challenging and requires domain expertise . |
| Approach: | They propose to build dialog policies with two agents as dialog agents to avoid building a user simulator beforehand. |
| Outcome: | The proposed method can build a system policy and a user policy simultaneously . it can achieve high task success rate through conversational interaction . |
Federated Learning of Gboard Language Models with Differential Privacy (2023.acl-industry)
Copied to clipboard
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan Mcmahan, Jesse Rosenstock, Yuanbo Zhang
| Challenge: | Using federated learning and differential privacy, we train and deploy language models with federation and DP in Google Keyboard. |
| Approach: | They train and deploy language models with federated learning and differential privacy in Google Keyboard . |
| Outcome: | The proposed algorithm achieves meaningfully formal DP guarantees without uniform sampling of clients. |
Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models learn non-trivial abstractions during pretraining, but it is not well understood when and how these specific linguistic abilities emerge. |
| Approach: | They propose a method to track the evolution of linguistic features during pretraining by using sparse crosscoders to discover and align features across model checkpoints. |
| Outcome: | The proposed approach can detect features emergence, maintenance, and discontinuation during training stages. |
Guided Knowledge Generation with Language Models for Commonsense Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved notable success in commonsense reasoning tasks, benefiting from extensive world knowledge acquired through extensive pretraining. |
| Approach: | They propose a method to generate knowledge explanations and to automatically assign labels based on the probability of correct answers. |
| Outcome: | The proposed method outperforms baselines on four widely-used commonsense reasoning benchmarks and shows that it can generate high quality knowledge leading to correct answers. |
Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to perform named entity recognition (NER) on unlabeled data are difficult to obtain in low-resource domains. |
| Approach: | They propose ways to use unlabeled data for pretraining to improve performance in downstream tasks. |
| Outcome: | The proposed methods outperform models trained on unlabeled data on seven domains. |
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages (2023.acl-long)
Copied to clipboard
Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, Hinrich Schütze
| Challenge: | Lack of LLMs supporting low-resource languages is a serious impediment to bringing NLP to all of the world. |
| Approach: | They create a model that scales LLMs horizontally and a corpus that covers 511 low-resource languages. |
| Outcome: | The proposed model improves on five diverse tasks across low- and high-resource languages. |
RadLing: Towards Efficient Radiology Report Understanding (2023.acl-industry)
Copied to clipboard
| Challenge: | a few pre-trained language models have produced comparable results in fine-tuning tasks in radiology domain. |
| Approach: | They propose a continuously pretrained language model with ELECTRA-small architecture that can compete with state-of-the-art results in radiology domain. |
| Outcome: | The proposed model can compete with state-of-the-art models for fine tuning tasks in radiology domain. |
Only for the Unseen Languages, Say the Llamas: On the Efficacy of Language Adapters for Cross-lingual Transfer in English-centric LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Most state-of-the-art large language models (LLMs) are trained mainly on English data, limiting their effectiveness on non-English, especially low-resource, languages. |
| Approach: | They train language adapters for 13 languages and evaluate their effectiveness on downstream tasks using either task adapters or in-context learning. |
| Outcome: | The proposed language adapters improve performance for languages not seen during pretraining, but provide negligible benefit for seen languages. |
Natural Context Drift Undermines the Natural Language Understanding of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | generative Large Language Models (LLMs) are based on natural text evolution . |
| Approach: | They propose a framework for curating naturally evolved variants of reading passages from contemporary QA benchmarks and for analysing LLM performance across a range of semantic similarity scores. |
| Outcome: | The proposed framework evaluates QA datasets and LLMs with publicly available training data. |
DeepStruct: Pretraining of Language Models for Structure Prediction (2022.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models perform structural understanding tasks that focus on understanding one aspect of the text. |
| Approach: | They propose a method for improving the structural understanding abilities of language models by pretraining them to generate structures from the text on task-agnostic corpora. |
| Outcome: | The proposed model performs state-of-the-art on 21 of 28 datasets. |
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering (2022.naacl-main)
Copied to clipboard
| Challenge: | a table-based question answering system requires complex reasoning and alignment between questions and tables. |
| Approach: | They propose a table-based QA model that consumes both natural and synthetic data . they combine retrieval with masking to pair natural sentences with QA . |
| Outcome: | The proposed model outperforms existing models in few-shot and full settings and on WikiTableQuestions. |
CompAct: Compressed Activations for Memory-Efficient LLM Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on reducing peak memory utilization on GPUs, but most work only target the computation graph during training. |
| Approach: | They propose a technique that reduces peak memory utilization on GPUs by 25-30% for pretraining and 50% for fine-tuning of LLMs. |
| Outcome: | The proposed technique reduces peak memory utilization on GPUs by 25-30% for pretraining and 50% for fine-tuning of LLMs. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
Controlling Language Confusion in Multilingual LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages. |
| Approach: | They propose a supervised fine-tuning methodology which optimizes the likelihood of correct tokens without explicitly penalizing undesired outputs such as cross-lingual mixing. |
| Outcome: | The proposed model suppresses language-confused generation while maintaining strong language consistency even under high decoding temperatures while preserving general QA performance. |
Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Strong pretraining approaches for grammatical error correction require extensive use of a pseudo-parallel corpus. |
| Approach: | They propose to use bidirectional and auto-regressive transformers as a generic pretrained encoder-decoder model for grammatical error correction (GEC) they find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC. |
| Outcome: | The proposed model achieves comparable results to the current strong results in English GEC. |
Measuring Progress in Fine-grained Vision-and-Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | X-VLM models lack "fine-grained" understanding of relationships, verbs and numbers in images . pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language tasks . |
| Approach: | They investigate models that outperform other baselines on fine-grained data . they highlight importance of novel losses and rich data sources for learning fine-grain skills . |
| Outcome: | The proposed model outperforms baseline models on four fine-grained benchmarks . the model outpersforms other baseline models and even degrades performance . |
Planning with Learned Entity Prompts for Abstractive Summarization (2021.tacl-1)
Copied to clipboard
| Challenge: | a simple but flexible mechanism is used to ground the generation of abstractive summaries. |
| Approach: | They propose a mechanism to learn an intermediate plan to ground the generation of abstractive summaries. |
| Outcome: | The proposed model outperforms state-of-the-art methods for faithfulness on CNN and BillSum. |
Towards Visual Question Answering on Pathology Images (2021.acl-short)
Copied to clipboard
| Challenge: | Pathology imaging is used for identifying the causes and effects of diseases or injuries. |
| Approach: | They propose a pathological visual question answering framework to analyze pathology images and answer medical questions related to these images. |
| Outcome: | The proposed framework performs self-supervised pretraining and finetuning end-to-end to learn powerful visual and textual representations jointly and automatically identifies and excludes noisy self-controlled examples from pretraining. |
Acrostic Poem Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Acrostic poems contain a hidden message; typically, the first letter of each line spells out a word or short phrase. |
| Approach: | They propose a task for acrostic poem generation in English with multiple constraints . they define the task as a generation task with multiple constraint constraints based on a conditional neural language model and a neural rhyming model . |
| Outcome: | The proposed task is based on a baseline model and a neural rhyming model. |
Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning (2025.acl-short)
Copied to clipboard
| Challenge: | Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. |
| Approach: | They propose to use image captions to implicitly align text representations between languages to make them usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
| Outcome: | The proposed approach is usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
SERENGETI: Massively Multilingual Language Models for Africa (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretrained models acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. |
| Approach: | They develop a set of massively multilingual language models that covers 517 African languages and language varieties. |
| Outcome: | The proposed models outperform 4 models that cover 4-23 African languages on eight natural language understanding tasks, achieving 82.27 average F_1. |
Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing researches in Scene Graph Generation (SGG) focus on refining model architectures that are trained from scratch with datasets like Visual Genome or Open Images. |
| Approach: | They propose to integrate pretrained Vision-language Models into SGG to improve representation by estimating the unattainable predicates distribution. |
| Outcome: | The proposed method significantly improves the performance of the debiased VLMs with SGG models. |
VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to “vet” labels from noisy captions for weakly-supervised object detection are limited for object detection. |
| Approach: | They propose a technique to “vet” labels extracted from noisy captions and use them for weakly-supervised object detection without any bounding boxes. |
| Outcome: | The proposed method improves WSOD without label vetting by 30% on five datasets. |
On the Transferability of Minimal Prediction Preserving Inputs in Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work establishes the presence of short, uninterpretable input fragments that yield high confidence and accuracy in neural models. |
| Approach: | They investigate competing hypotheses for the existence of MPPIs in question answering . they discover a perplexing invariance of MPIs to random training seed, model architecture, pretraining, and training domain. |
| Outcome: | The proposed model performance is higher than comparable short queries. |
POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection (2022.findings-naacl)
Copied to clipboard
| Challenge: | a lack of general-purpose tools to characterize and predict ideology across genres of text remains a challenge . a recent study compared ideology-driven pretraining tasks with long or formal written texts . |
| Approach: | They propose to use a large-scale dataset to train pretraining models that compare political news articles on the same story written by different ideologies. |
| Outcome: | The proposed model outperforms baseline models and state-of-the-art models on ideology prediction and stance detection tasks. |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. |
| Approach: | They propose a synthetic task, Multilingual Othello, as a testbed to investigate the factors that contribute to the learning of a language-neutral representation. |
| Outcome: | The proposed approach induces the learning of language-neutral representation and facilitates cross-lingual transfer. |
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
Bigger Data or Fairer Data? Augmenting BERT via Active Sampling for Educational Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) encode bias against protected groups in the representations they learn, which may harm the prediction fairness of downstream models. |
| Approach: | They propose to quantify the awareness that a pretrained language model (BERT) has regarding people’s protected attributes and augment it to enhance prediction fairness of downstream models. |
| Outcome: | The proposed method improves fairness and accuracy of models by inhibiting the awareness of protected attributes in the PLMs. |
Few-Shot Semantic Parsing for New Predicates (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study shows that state-of-the-art neural semantic parsers are less accurate when there is only a handful of utterance-logical form pairs per predicate. |
| Approach: | They propose to use a meta-learning method to train a few-shot learning problem . they also propose to regularize attention scores with alignment statistics and apply a smoothing technique . |
| Outcome: | The proposed method outperforms baselines in one and two-shot settings. |
Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit. |
| Approach: | They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance. |
| Outcome: | The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI. |
Large Language Models with Controllable Working Memory (2023.findings-acl)
Copied to clipboard
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, Sanjiv Kumar
| Challenge: | Large language models (LLMs) have led to a series of breakthroughs in natural language processing due to the massive amounts of world knowledge they memorize during pretraining. |
| Approach: | They propose a method to inject counterfactual and irrelevant contexts into standard supervised datasets to strengthen both controllability and robustness. |
| Outcome: | The proposed method improves controllability and robustness across model architectures and sizes. |
Planning and Generating Natural and Diverse Disfluent Texts as Augmentation for Disfluency Detection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to disfluency detection heavily depend on labeled data. |
| Approach: | They propose a Planner-Generator based disfluency generation model that generates natural disfluent texts as augmented data. |
| Outcome: | The proposed model outperforms baselines and leads to state-of-the-art performance on Switchboard corpus. |
Tracing Multilingual Factual Knowledge Acquisition in Pretraining (2025.findings-emnlp)
Copied to clipboard
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, Hinrich Schuetze
| Challenge: | Large Language Models are capable of recalling multilingual factual knowledge, but most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency unexplored. |
| Approach: | They trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study. |
| Outcome: | The results show that fact frequency is the key to a better recall of multilingual facts, regardless of language, and some low-frequency facts in non-English languages can still be correctly recalled. |
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
Enhancing Multimodal Unified Representations for Cross Modal Generalization (2025.findings-acl)
Copied to clipboard
Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, Zhou Zhao
| Challenge: | Existing studies on discrete unified representations overlook important distinctions between different dimensions of features. |
| Approach: | They propose to use a codebook to optimize unified representations from pretraining and fine- and coarse-grained disentangling to optimize the representations. |
| Outcome: | The proposed methods improve the interpretability of multimodal unified representations . they use training-free optimization of codebook and fine and coarse cross-modal disentangling . |
AWARE: Agentic Knowledge Warehousing for Contextual Intelligence (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in information seeking tasks, but their knowledge is limited in coverage and timeliness. |
| Approach: | They propose an agentic knowledge warehousing framework that transforms unstructured data into minimal, task-conditioned knowledge representations consumable by LLMs. |
| Outcome: | Experiments on GAIA, WebWalker, and BrowseComp-Plus show improvements over baselines. |
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on how to encode compositional task structure has been limited by semantic parsing and multihop reasoning for the purpose of Q&A. |
| Approach: | They propose an approach to decomposing a target task into component tasks and fine-tuning smaller LMs on a curriculum of such component tasks. |
| Outcome: | The proposed approach outperforms end-to-end learning even with equal data, and gets better as more component tasks are modeled. |
PEFTDebias : Capturing debiasing information using PEFTs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research suggests that biases acquired during pretraining can propagate to downstream models, resulting in superficial text dependencies and potential implicit bias. |
| Approach: | They propose a parameter-efficient fine-tuning approach to mitigate implicit biases within foundation models by incorporating parameters into the model and freezing them during the fine-uning process. |
| Outcome: | The proposed method reduces biases in foundation models by incorporating parameters and freezing them during fine-tuning. |
“Chère maison” or “maison chère”? Transformer-based prediction of adjective placement in French (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prior work has shown that transformer-based language models are insensitive to permutated word order, but this is not the case with pretraining. |
| Approach: | They conduct experiments to assess whether transformer-based language models are able to learn the adjective position in noun phrases in French. |
| Outcome: | The proposed model is weaker with complex structures and fixed expressions, but favors context and global syntactic roles. |
Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for ICD coding are limited due to the high-dimensional space of multi-label assignment and the long-tail challenge. |
| Approach: | They propose a prompt-based fine-tuning technique with label semantics to solve this challenge. |
| Outcome: | The proposed method outperforms state-of-the-art methods on a benchmark dataset of code assignment in 14.5% of cases. |
Applications of BERT Models Towards Automation of Clinical Coding in Icelandic (2024.findings-naacl)
Copied to clipboard
| Challenge: | Traditionally, clinical coding is manual and laborintensive task prone to human error. |
| Approach: | They analyze 25 years of electronic health records from the Landspitali University Hospital in Icelandic to explore the potential of using NLP for clinical coding. |
| Outcome: | The best-performing model achieves competitive results in micro and macro F1 scores, with label attention contributing significantly to its success. |
DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility (2025.findings-naacl)
Copied to clipboard
| Challenge: | Video-to-speech (V2S) synthesis requires acoustic hints to accurately reconstruct both speech content and speaker characteristics from video clips alone. |
| Approach: | They propose a video-to-speech (V2S) model that predicts Mel-spectrograms directly from video frames. |
| Outcome: | The proposed model outperforms existing models in acoustic intelligibility and preserves speaker-specific characteristics. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
Diversity-Aware Coherence Loss for Improving Neural Topic Models (2023.acl-short)
Copied to clipboard
| Challenge: | Experimental results show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters. |
| Approach: | They propose a variational autoencoder framework that minimizes the posterior and prior divergence and a diversity-aware coherence loss that encourages the model to learn corpus-level coherency scores while maintaining high diversity between topics. |
| Outcome: | The proposed approach significantly improves the performance of neural topic models without pretraining or additional parameters. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Vision-Language Models Align with Human Neural Representations in Concept Processing (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based vision-language models capture the multimodality of concept processing in the human brain. |
| Approach: | They analysed multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. |
| Outcome: | The transformer-based vision-language models outperform language-only models in two experimental conditions, while only some outperformed the language-based models. |
OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages (2022.acl-long)
Copied to clipboard
| Challenge: | a new study examines the performance of pretraining for sign language recognition in low-resource settings. |
| Approach: | They propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference. |
| Outcome: | The proposed model reduces training time and allows efficient inference in sign languages. |
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training (2025.emnlp-industry)
Copied to clipboard
Maryam Dialameh, Rezaul Karim, Hossein Rajabzadeh, Omar Mohamed Awad, Boxing Chen, Hyock Ju Kwon, Walid Ahmed, Yang Liu
| Challenge: | ECHO-LLaMA transforms LLa MA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. |
| Approach: | They propose an efficient LLaMA architecture that transforms LLama models into shared KV caching across certain layers, reducing computational complexity while maintaining or improving language performance. |
| Outcome: | ECHO-LLaMA achieves up to 77% higher token-per-second throughput during training, up to 16% higher Model FLOPs Utilization (MFU) and up to 14% lower loss when trained on an equal number of tokens. |
Understanding the Effect of Model Compression on Social Bias in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text, leading to representational harm. |
| Approach: | They propose to use quantization and knowledge distillation to reduce the computational burden of LLMs to mitigate the effects of inappropriate social biases learned during pretraining. |
| Outcome: | The proposed methods reduce the computational burden of large language models by reducing their size and complexity. |
NeighXLM: Enhancing Cross-Lingual Transfer in Low-Resource Languages via Neighbor-Augmented Contrastive Pretraining (2025.findings-emnlp)
Copied to clipboard
| Challenge: | NeighXLM is a neighbor-augmented contrastive pretraining framework . it exploits intra-language semantic relationships captured during pretraining to construct high-quality positive pairs. |
| Approach: | They propose a neighbor-augmented contrastive pretraining framework that mines semantic neighbors from unlabeled corpora to enrich target-language supervision. |
| Outcome: | The proposed framework enriches target-language supervision by mining semantic neighbors from unlabeled corpora. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
Japanese Zero Anaphora Resolution Can Benefit from Parallel Texts Through Neural Transfer Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Using a pretraining model, we find that the performance of Japanese zero anaphora resolution (ZAR) is improved by using machine translation. |
| Approach: | They propose to inject machine translation as an intermediate task between pretraining and ZAR by injecting machine translation into a pretrained BERT model and injecting it into MT. |
| Outcome: | The proposed framework shows that Japanese zero anaphora resolution (ZAR) can be improved by transfer learning from machine translation (MT). |
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing abstractive summarization models ignore abundant unlabeled corpora resources . TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets . |
| Approach: | They propose a transformer-based unsupervised text summarization system with pretraining on large-scale data. |
| Outcome: | The proposed system outperforms baseline models on NYT, CNN/DM and English Gigaword datasets with various document styles. |
TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can produce informative and fluent text with the help of large-scale datasets, but they suffer insufficient learning problem with limited training data. |
| Approach: | They propose to use table transformation module with template to rewrite structured table in natural language as input for GPT-2 and exploit multi-task learning with two auxiliary tasks to preserve table’s structural information. |
| Outcome: | The proposed model outperforms existing systems on most few-shot settings. |
MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretraining is becoming the norm in Vision-Language (VL) modeling. |
| Approach: | They propose a method for augmenting generative language models with additional modalities using adapter-based finetuning. |
| Outcome: | The proposed method outperforms Frozen on open-ended generative tasks while maintaining the language model weights. |
Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) |
| Approach: | They propose to use LLMs to probe their behavior using controlled experiments. |
| Outcome: | The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do. |
Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing open-source LLMs exhibit limited effectiveness in processing Vietnamese . lack of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation exacerbates these issues. |
| Approach: | They propose to fine tune LLMs specifically for Vietnamese and develop a framework for evaluation . they find that larger models introduce more biases and uncalibrated outputs . |
| Outcome: | The proposed framework finetunes LLMs specifically for Vietnamese and provides a framework for evaluation . |
Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models (2023.eacl-main)
Copied to clipboard
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer
| Challenge: | Pretrained language models store a large amount of factual information that can be elicited by prompting or finetuning. |
| Approach: | They propose methods to measure model factual beliefs and update incorrect beliefs in models . they propose a new visualization tool that shows relationships between stored model beliefs . |
| Outcome: | The proposed methods improve models' consistency and accuracy, the authors show . their methods outperform existing methods in more difficult settings, the paper shows . |
Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024.naacl-long)
Copied to clipboard
| Challenge: | masked language modeling is widely used as a pretraining component in Vision and language (V+L) but performance on benchmarks has not received the attention it deserves. |
| Approach: | They propose a curriculum masking scheme that uses a parallel mask selection agent to mask tokens at a frequency proportional to the level of cross modal interaction necessary to reconstruct them. |
| Outcome: | The proposed method improves relational understanding on a wide range of V+L tasks. |
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Multilingual AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on generating text from structured data into English has focused on bridging the gap between structure and natural language (NL) and semantically underspecified input and fully specified output. |
| Approach: | They propose a multilingual approach that can decode into 21 different languages . they leverage advances in cross-lingual embeddings and pretraining to generate multilingual models . |
| Outcome: | The proposed model surpasses baselines that generate into one language in eighteen languages. |
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a large number of pretraining corpora are not publicly available, and it is unclear how much foreign language data exists in monolingual models. |
| Approach: | They propose to use English pretraining corpora to analyze their language composition . they find that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |
| Outcome: | The proposed model is not truly monolingual when pretrained at scale, the authors show . they show that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal . |
| Approach: | They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities. |
| Outcome: | The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. |
Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models lack contextual knowledge, resulting in text with factual inconsistencies or contextually unfaithful content. |
| Approach: | They propose a method that integrates contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. |
| Outcome: | The proposed method improves context grounding during generation without training. |
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC (N19-1)
Copied to clipboard
| Challenge: | In response to this development, there have been a flurry of new datasets for question answering. |
| Approach: | They propose to use SQuAD 2.0, QuAC, and CoQA to provide question answering on textual data. |
| Outcome: | The proposed datasets provide complementary coverage of the first two aspects, but weak coverage of third. |
PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | A wide range of NLP tasks benefit from fine-tuning of pretrained language models (PLMs), however, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine- tuned model. |
| Approach: | They propose a noisy training mechanism which considers each parameter’s importance in the downstream task to help fine-tune pretrained language models. |
| Outcome: | The proposed method can be used to fine-tune pretrained language models on a wide range of tasks and consistently achieve higher performance. |
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing open-domain question answering systems are insufficient to capture deep semantic matching that goes beyond lexical overlaps. |
| Approach: | They propose a sample-efficient method to pretrain the paragraph encoder using an existing pretraining model instead of heuristically created pseudo question-paragraph pairs. |
| Outcome: | The proposed method outperforms a strong dense retrieval baseline that uses 6 times more computation for training. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
VIMI: Grounding Video Generation through Multi-modal Instruction (2024.emnlp-main)
Copied to clipboard
Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Tsai-Shien Chen, Kuan-Chieh Wang, Ivan Skorokhodov, Graham Neubig, Sergey Tulyakov
| Challenge: | Existing text-to-video diffusion models rely on text-only encoders for their pretraining, restricting their versatility and application in multimodal integration. |
| Approach: | They propose a multimodal conditional video generation framework for pretraining on augmented text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within a model. |
| Outcome: | The proposed model can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control. |
Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt tuning is a technique for adapting large-scale pretrained language models for downstream tasks. |
| Approach: | They propose to condition a frozen pretrained language model with soft prompts from data . they propose to use a domain adaptation technique to regularize the decision boundary . |
| Outcome: | The proposed method outperforms full-model tuning in data-scarce settings by a large margin. |
Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs (2026.eacl-long)
Copied to clipboard
Alireza Dehghanpour Farashah, Aditi Khandelwal, Marylou Fauchard, Zhuan Shi, Negar Rostamzadeh, Golnoosh Farnadi
| Challenge: | Existing studies on unlearning in multilingual large language models focus on monolingual settings, typically English. |
| Approach: | They propose to use a multilingual data and concept unlearning model to investigate the problem . they extend benchmarks for factual knowledge and stereotypes into ten languages . |
| Outcome: | The proposed model is able to unlearning in 10 languages across five languages and resource levels. |
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. |
| Approach: | They propose a position encoding strategy that uses random float sampling to generalize to unseen lengths. |
| Outcome: | The proposed strategy can generalize to lengths unseen during training and in benchmarks. |
ConVEx: Data-Efficient and Few-Shot Slot Labeling (2021.naacl-main)
Copied to clipboard
| Challenge: | ConVEx is an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. |
| Approach: | They propose an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks that uses a pairwise cloze task and reddit data. |
| Outcome: | The proposed approach is well aligned with its intended use on slot-labeling tasks and can be used across a range of domains and data sets. |
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation models. |
| Approach: | They propose to use knowledge distillation to generate training data for non-autoregressive translation models by leveraging pretraining. |
| Outcome: | The proposed approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets. |
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models . |
| Approach: | They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks. |
| Outcome: | The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data. |
Does Pretraining for Summarization Require Knowledge Transfer? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing theories claim that pretraining models learn linguistic knowledge from the pretraining corpus, but scientific explanations for these benefits remain unknown. |
| Approach: | They propose to use random character n-grams to test models on real corpora to see if the small residual benefit of using real data could be accounted for by the structure of the pretraining task. |
| Outcome: | The proposed task performs on documents consisting of character n-grams, whereas pretrained models perform on real corpora with no residual benefit. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
Domain Adversarial Fine-Tuning as an Effective Regularizer (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing fine-tuning techniques can degrade general-domain representations . however, fine-timing can lead to catastrophic forgetting of knowledge . |
| Approach: | They propose a new regularization technique that complements the task-specific loss used during fine-tuning with an adversarial objective. |
| Outcome: | Empirical results show that AFTER improves performance on various natural language understanding tasks compared to standard fine-tuning. |
Cross-lingual Cross-modal Pretraining for Multimodal Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English. |
| Approach: | They propose a new approach to learn cross-lingual cross-modal representations for matching images and captions in multiple languages using an annotated corpus. |
| Outcome: | The proposed model achieves impressive performance on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions. |
Modeling Coreference Relations in Visual Dialog (2021.eacl-main)
Copied to clipboard
| Challenge: | Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. |
| Approach: | They propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way based on linguistic knowledge and discourse features of human dialog. |
| Outcome: | The proposed model achieves state-of-the-art performance on the VisDial v1.0 dataset without pretraining on other vision language datasets. |
Your Transformer is Secretly Linear (2024.acl-long)
Copied to clipboard
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)
Copied to clipboard
Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy
| Challenge: | Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. |
| Approach: | They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning. |
| Outcome: | The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance. |
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained neural language models (LMs) are prone to generating toxic language . we investigate the extent to which pretrained LMs can be prompted to generate toxic language. |
| Approach: | They propose to use a dataset to test the extent to which pretrained LMs can generate toxic language . they use toxicity-based text generation algorithms to generate toxic text from natural prompts . |
| Outcome: | The proposed model can degenerate into toxic text even from seemingly innocuous prompts. |
Multilingual Multimodal Learning with Machine Translated Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, most vision-and-language pretraining research focuses on English tasks due to the availability of datasets. |
| Approach: | They propose a framework for machine translating English multimodal data to improve training data . they propose two metrics to prevent models from learning from low-quality translated text . |
| Outcome: | The proposed framework can be applied to any multimodal dataset and model. |
MonoByte: A Pool of Monolingual Byte-level Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models can achieve zero-shot cross-lingual performance on various NLP tasks, but due to the cost of pretraining, they often use public models with limited budgets. |
| Approach: | They propose to use tokenized models to test cross-lingual ability in multilingual and monolingual corpora. |
| Outcome: | The results show that models pretrained on multilingual and even monolingual corpora perform better than models pre-trained on SOTA models. |
ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Language Models excel in understanding textual descriptions of proteins, but struggle to process texts. |
| Approach: | They propose a framework for Protein-to-Text Generation for Text-based Protein Understanding that integrates a PLM as its protein understanding module. |
| Outcome: | The proposed framework surpasses existing baselines and is highly efficient in protein-to-text generation. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
Language model acceptability judgements are not always robust to context (2023.acl-long)
Copied to clipboard
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
| Challenge: | a recent study found that models prefer acceptable inputs over acceptable ones. |
| Approach: | They find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but unstable when contexts match the test stimuli in syntactic structure. |
| Outcome: | The proposed model performance improves when contexts match syntactic structure, and declines when they are unacceptable. |
Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level (2021.findings-acl)
Copied to clipboard
| Challenge: | Larger models have higher out-of-distribution robustness, while smaller models have lower accuracy on rare subgroups. |
| Approach: | They develop statistically rigorous methods to investigate whether large models are better on every instance . they find that individual predictions are highly sensitive to noise in the randomness in training . |
| Outcome: | The proposed model is worse than BERT-MINI on 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%. |
Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly (2024.naacl-long)
Copied to clipboard
| Challenge: | Current large language models show imbalance abilities in different languages . authors propose two approaches to improve cross-lingual knowledge alignment . |
| Approach: | They propose a framework to assess cross-lingual knowledge alignment of large language models . they propose multilingual pretraining and multilingual instruction tuning to address this problem . |
| Outcome: | The proposed framework assesses the cross-lingual knowledge alignment of LLMs in performance, consistency and conductivity levels. |
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have been successful when finetuned to downstream tasks . however, it is difficult to determine whether the knowledge that finetuning LMs contain is learned during the pretraining or the finetailing process. |
| Approach: | They propose a method to create prompts for a diverse set of tasks using a gradient-guided search. |
| Outcome: | The proposed method performs sentiment analysis and natural language inference without additional parameters and finetuning. |
Generative-to-Discriminative Test-Time Adaptation via Manifold-Aware Diffusion and Bayesian Distillation (2026.findings-acl)
Copied to clipboard
Boyun Zhang, Zequn Xie, Fangming Feng, Zihan Zhang, Yongbo He, Chuxin Wang, Sihang Cai, Tao Jin, Qifei Zhang
| Challenge: | Existing discriminative approaches suffer from "confident but wrong" failure mode, blindly adapting to OOD noise leading to error accumulation. |
| Approach: | They propose a TTA framework that harmonizes the robustness of generative diffusion models with the efficiency of discriminative regression networks via Bayesian Diffusion Distillation (BDD). |
| Outcome: | The proposed framework reduces MAE from 0.6872 to 0.5673 and boosts binary accuracy by 5.81 percentage points (reaching 57.33%) it also reduces the MAE of the MOSI to SIMS shift and achieves an 11.18-point gain over the baseline. |
How to Adapt Your Pretrained Multilingual Model to 1600 Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models perform best for languages seen during pretraining . methods exist to improve performance for unseen languages, but have been evaluated using amounts of raw text only available for a small fraction of the world’s languages. |
| Approach: | They evaluate the performance of existing methods to adapt pretrained multilingual models to new languages using a resource available for close to 1600 languages: the New Testament. |
| Outcome: | The proposed models perform best for languages seen during pretraining . the results show that the most efficient approach is simplest and the most accurate . |
On Biasing Transformer Attention Towards Monotonicity (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work has focused on learning monotonic attention behavior via specialized attention functions or pretraining. |
| Approach: | They introduce a monotonicity loss function compatible with standard attention mechanisms and test it on sequence-to-sequence tasks. |
| Outcome: | The proposed monotonicity loss function can achieve largely monotonic behavior on grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization tasks. |
Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | This study examines the ability of Large Language Models to encapsulate cultural nuances across diverse linguistic landscapes. |
| Approach: | They examine the efficacy of language-specific instruction tuning and the impact of pretraining on dominant language data in Large Language Models. |
| Outcome: | The findings highlight a nuanced landscape, with inconsistencies and biases, particularly in non-Western cultures. |
EfficientLLM: Unified Pruning-Aware Pretraining for Auto-Designed Compact Language Models (2026.acl-long)
Copied to clipboard
Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Haokun Lin, Xianlin Zeng, Guoqi Li, Jiajun Zhang
| Challenge: | Large language models (LLMs) driven by scaling laws can be developed in large model sizes. |
| Approach: | They propose a pruning-aware pretraining approach that decouples LLM pruning from direct pretraining. |
| Outcome: | The proposed model outperforms pretraining models with 100M 1B parameters in commen sense benchmarks. |
From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experiments show that models trained on multi-way parallel data outperform those trained on unaligned data. |
| Approach: | They propose a large-scale, high-quality multi-way parallel corpus based on TED Talks that spans 113 languages with up to 50 languages aligned in parallel. |
| Outcome: | The proposed model outperforms models trained on unaligned multilingual data on six multilingual benchmarks. |
An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021.findings-emnlp)
Copied to clipboard
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Existing models for dialogue summarization focus on extracting the main events of short conversations, but real-world dialogues are difficult to train. |
| Approach: | They propose three strategies to deal with the lengthy input problem and locate relevant information using long dialogue datasets. |
| Outcome: | The retrieve-then-summarize pipeline models yield the best performance on three long dialogue datasets. |
Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | cuneiform clay tablets were written in 2500 BCE - 100 CE and are a target of extensive transcription and transliteration efforts due to their deterioration. |
| Approach: | They propose to use a masked language modelling task to complete missing text given cuneiform clay tablets written on cuniform signswedges (2500 BCE - 100 CE) they develop models which automatically complete these missing signs based on contextual cues and greedy decoding schemes. |
| Outcome: | The proposed models perform well on missing token prediction (89% hit@5) despite data scarcity (1M tokens), and human evaluations show that they are able to transcribe texts in extinct languages. |
Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Text (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for named entity disambiguation are limited by coarse-grained structural resources in biomedical knowledge bases and training datasets that provide low coverage over uncommon resources. |
| Approach: | They propose a method that integrates structural knowledge from general text knowledge bases to the medical domain. |
| Outcome: | The proposed method improves disambiguation accuracy on two benchmark medical NED datasets by up to 57 points. |
Task-adaptive Pre-training of Language Models with Word Embedding Regularization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models acquire domain-independent knowledge through pre-training with massive textual resources. |
| Approach: | They propose a task-adaptive pre-training process that makes static embeddings close to the word embedds obtained in the target domain. |
| Outcome: | The proposed process improves on BioASQ and SQuAD when the pre-training corpora were not dominated by indomain data. |
SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task. |
| Approach: | They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines . |
| Outcome: | The proposed model improves sample-efficiency and performance over strong baselines. |
Sustainable Modular Debiasing of Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing debiasing methods modify all of the PLM parameters, which is costly and leads to (catastrophic) forgetting of useful language knowledge. |
| Approach: | They propose a modular debiasing approach based on dedicated adapters that inject adapter modules into the original PLM layers and update only the adapters. |
| Outcome: | The proposed approach is based on dedicated adapters and retains fairness even after large-scale training. |
Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets on spatial language processing are either synthetic or at small scale. |
| Approach: | They propose a dataset for transfer learning on spatial question answering and spatial role labeling that includes a larger variety of spatial relation types and spatial expressions. |
| Outcome: | The proposed dataset can be used to evaluate spatial language processing models in real-world situations. |
Measuring Cross-lingual Transfer in Bytes (2024.naacl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models can transfer knowledge to target languages with minimal or no examples . underlying mechanisms for this transfer remain unclear, with hypotheses ranging from language contamination to syntactic similarity. |
| Approach: | They conducted an experiment to investigate whether multilingual models transfer knowledge to target languages . they found that models initialized from diverse languages perform similarly to a target language . |
| Outcome: | a new study shows that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. |
UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining (2022.acl-long)
Copied to clipboard
| Challenge: | Existing phrase representation learning methods combine unigram representations in a context-free manner or rely on extensive annotations to learn context-aware knowledge. |
| Approach: | They propose a novel unsupervised contrastive learning framework for context-aware phrase representations and topic mining. |
| Outcome: | The proposed framework outperforms the state-of-the-art phrase representation model by 38.2% NMI on four entity clustering tasks. |
On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining. |
| Approach: | They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss. |
| Outcome: | The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures. |
Automatic Correction of Writing Anomalies in Hausa Texts (2026.acl-long)
Copied to clipboard
| Challenge: | Hausa texts are often characterized by writing anomalies such as incorrect character substitutions and spacing errors, which hinder natural language processing (NLP) applications. |
| Approach: | They propose to fine tune transformer-based Hausa-based models to correct writing anomalies by introducing synthetically generated noise to mimic real-world errors. |
| Outcome: | The proposed model improves Hausa text quality and improves other low-resource languages. |
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages (2022.acl-long)
Copied to clipboard
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, Katharina Kann
| Challenge: | Pretrained multilingual models can perform cross-lingual transfer in a zero-shot setting, even for unseen languages. |
| Approach: | They propose to extend XNLI to 10 indigenous languages of the Americas and test multiple zero-shot and translation-based approaches. |
| Outcome: | The proposed model can perform cross-lingual transfer in a zero-shot setting even for languages unseen during pretraining. |
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2022.naacl-main)
Copied to clipboard
| Challenge: | a pretrained model is optionally adapted through domain-specific pretraining, followed by task-specific finetuning. |
| Approach: | They establish a suite of eight tasks across different domains to quantify the effects of temporal misalignment in modern NLP systems. |
| Outcome: | The proposed tasks are based on eight domains and periods of time spanning five years or more and show that they have stronger effects than previous studies. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning can tackle zero-shot and few-shot NLP tasks . authors propose a method that makes use of pre-trained language models . |
| Approach: | They propose to map NLP tasks into natural language prompts, which are then filled by pre-trained language models. |
| Outcome: | The proposed method outperforms standard prompt-based methods in few-shot settings. |
Revisiting Grammatical Error Correction Evaluation and Beyond (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining-based (PT) evaluation metrics are not effective for training grammatical error correction systems. |
| Approach: | They propose a pretraining-based GEC evaluation metric which only uses PT-based metrics to score the corrected parts of the system. |
| Outcome: | The proposed evaluation metric outperforms existing methods on a CoNLL14 evaluation task. |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2020 focus on semi-structured document classification in a zero-shot setting . positional, layout, and style information play a vital role in interpreting such documents . |
| Approach: | They propose a matching-based approach that relies on a pairwise contrastive objective for pretraining and fine-tuning. |
| Outcome: | The proposed method significantly improves Macro F1 in the zero-shot learning setting. |
Probing LLMs for Joint Encoding of Linguistic Categories (2023.findings-emnlp)
Copied to clipboard
Giulio Starace, Konstantinos Papakostas, Rochelle Choenni, Apostolos Panagiotopoulos, Matteo Rosati, Alina Leidinger, Ekaterina Shutova
| Challenge: | Existing research suggests that a linguistic hierarchy emerges in large language models . little is known about how encodings of different linguistic phenomena interact within the models - and to what extent processing of linguistically-related categories relies on the same, shared model representations. |
| Approach: | They propose a framework for testing the joint encoding of linguistic categories in large language models. |
| Outcome: | The proposed framework shows that the same patterns hold across languages in multilingual LLMs. |
Soft Syntactic Reinforcement for Neural Event Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent event extraction methods rely on pre-trained language models but still suffer from errors due to a lack of syntactic knowledge. |
| Approach: | They propose a method to incorporate syntactic information into PLM-based models for event extraction (EE) this method uses a standard dependency corpus to select syntax-related dimensions of the model's representation. |
| Outcome: | The proposed method outperforms baseline models and existing syntactic reinforcement methods on sentence-level and document-level EE benchmark datasets. |
Towards Pareto-Efficient RLHF: Paying Attention to a Few High-Reward Samples with Reward Dropout (2024.findings-emnlp)
Copied to clipboard
| Challenge: | RLHF is a bi-objective problem that has the nature of a Pareto optimization . reward dropout is generalizable and most effective with non-pretrained target models . |
| Approach: | They propose a method that guarantees a Pareto improvement by leveraging reinforcement learning to fine-tune language models. |
| Outcome: | The proposed method guarantees a Pareto improvement on two benchmark datasets . it is generalizable and most effective with non-pretrained target models, saving the effort of pretraining. |
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)
Copied to clipboard
| Challenge: | PTLMs are used to extract knowledge from text on demand. |
| Approach: | They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task. |
| Outcome: | The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world. |
Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pretraining with an artificial language with nesting dependency structure provides some knowledge transferable to natural language. |
| Approach: | They propose to pretrain artificial languages with structural properties that mimic natural language and then test their performance on downstream tasks. |
| Outcome: | The proposed language models show strong performance across languages and languages. |
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models only use entity information in pretraining and do not explicitly use entities in downstream tasks. |
| Approach: | They propose to leverage Wikipedia entity representations for downstream tasks . they train a multilingual language model with 24 languages with entity representation . |
| Outcome: | The proposed model outperforms word-based models in cross-lingual transfer tasks. |
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has obtained strong zero-shot results by prompting language models. |
| Approach: | They propose a mining-based approach that uses regular expressions to mine labeled examples from unlabeled corpora and fine tune a pretrained model. |
| Outcome: | The proposed method outperforms prompting on a wide range of tasks when using comparable templates. |
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained machine learning models perpetuate and even amplify existing biases in data . this can result in unfair outcomes that ultimately impact user experience . |
| Approach: | They quantify bias amplification in pretraining and after fine-tuning on vision-and-language models. |
| Outcome: | The results show that pretrained models can perpetuate and even amplify biases in data without compromising performance. |
Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable advancements in specialized fields such as finance, law, and medicine. |
| Approach: | They propose to provide datasets covering all major training stages including pretraining, instruction fine-tuning, and reasoning distillation with cybersecurity-specific self-reflection data. |
| Outcome: | Extensive ablation studies show that LLMs acquire their knowledge during pretraining, while reasoning distillation leads to a 15% gain in security certification (CISSP). |
Data Augmentation for Low-Resource Keyphrase Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing works on keyphrase generation rely on large-scale annotated datasets, which are not easy to acquire. |
| Approach: | They propose to use full text to improve keyphrase generation in resource-constrained domains by using the full text of the articles to augment their methods. |
| Outcome: | The proposed methods improve both present and absent keyphrase generation on three datasets and show that they are cost-effective. |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
Knowledge-Selective Pretraining for Attribute Value Extraction (2023.findings-emnlp)
Copied to clipboard
Hui Liu, Qingyu Yin, Zhengyang Wang, Chenwei Zhang, Haoming Jiang, Yifan Gao, Zheng Li, Xian Li, Chao Zhang, Bing Yin, William Wang, Xiaodan Zhu
| Challenge: | Existing methods for AVE are limited on rare attributes due to poor generalization ability. |
| Approach: | They propose to leverage pretraining and transfer learning to address weaknesses in existing methods. |
| Outcome: | The proposed method achieves new state-of-the-art performance without pretraining on rare attributes with limited training resources. |
CTC-based Non-autoregressive Textless Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing direct speech-to-speech translation models require text supervision during training, which is not feasible for numerous unwritten languages. |
| Approach: | They propose a non-autoregressive (NAR) model that generates discrete units from the source speech and employs a unit-based vocoder to synthesize the target. |
| Outcome: | The proposed model achieves translation quality comparable to the autoregressive model while preserving up to 26.81 decoding speedup. |
RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to update or supplement large language models struggle under continuous knowledge drift. |
| Approach: | They propose a dynamic event benchmark and time-aware retrieval baseline that captures how knowledge evolves over time. |
| Outcome: | The proposed method enables systematic evaluation of model adaptation under continuous knowledge drift. |
Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | MULTI-EURLEX is a dataset for topic classification of EU legal documents . fine-tuning a multilingually pretrained model in a single source language leads to catastrophic forgetting of multilingual knowledge and poor zero-shot transfer to other languages. |
| Approach: | They propose to use the dataset as a testbed for zero-shot cross-lingual transfer to exploit annotated training documents in one language to classify documents in another language. |
| Outcome: | The proposed model can be used to classify EU legal documents in other languages without a single source language and retain multilingual knowledge. |
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs. |
| Approach: | They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories. |
| Outcome: | The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models. |
PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit reasonable multilingual abilities, despite predominantly English-centric pretraining. |
| Approach: | They propose a framework that establishes multilingual alignment prior to language model pretraining and preserves this alignment using a code-switching strategy during pretraining. |
| Outcome: | Experiments in a synthetic English to English-Clone setting show that PreAlign outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-linguistic knowledge application. |
Multimodal Reasoning with Multimodal Knowledge Graph (2024.acl-long)
Copied to clipboard
| Challenge: | Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. |
| Approach: | They propose a multimodal reasoning method that leverages multimodal knowledge graphs to learn rich and semantic knowledge across modalities. |
| Outcome: | The proposed method outperforms state-of-the-art models on multimodal question answering and multimodal analogy reasoning tasks while training on only a small fraction of parameters. |
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models (2025.acl-long)
Copied to clipboard
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, WangYan WangYan, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, Bryan Hooi
| Challenge: | Large language models face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. |
| Approach: | They propose a training strategy for extending the context window of LLMs including impactful token analysis, position index transformation, and training optimization strategies. |
| Outcome: | Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size. |
Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are mainly trained on English data and struggle with low-resource languages. |
| Approach: | They propose to add a new language to Llama to improve classification accuracy for Persian tasks by aligning representations through bilingual pretraining and instruction datasets. |
| Outcome: | The proposed model performs on generation and classification tasks with no adverse impact and sometimes even improvements on English tasks. |
Characterizing Mechanisms for Factual Recall in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models often integrate facts they memorized with new information that appears in a given context, causing competition within the model. |
| Approach: | They investigate distributional and mechanistic determinants of LM behavior in a dataset that queries for knowledge of world capitals . they use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer in the logits . |
| Outcome: | The proposed method can increase the rate of generating the in-context answer to 88% of the time by scaling up or down the value vector of individual attention heads at runtime. |
NeoQA: Evidence-based Question Answering with Generated News Events (2025.findings-acl)
Copied to clipboard
| Challenge: | Evaluating Retrieval-Augmented Generation (RAG) in large language models is challenging because benchmarks can quickly become stale. |
| Approach: | They propose a benchmark to evaluate Retrieval-Augmented Generation (RAG) in large language models (LLMs) using timelines and knowledge bases of fictional news events and entities to prevent LLMs from leveraging pretraining knowledge. |
| Outcome: | The proposed benchmark prevents LLMs from leveraging pretraining knowledge and ensures that no prior evidence exists in their training data. |
The language of prompting: What linguistic properties make a prompt successful? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pretraining and instruction-tuned LLMs can achieve impressive performance on a multitude of tasks. |
| Approach: | They propose to use a standard for prompting research to better understand linguistic properties of LLMs. |
| Outcome: | The proposed standard would improve the performance of pre-trained and instruction-tuned LLMs on a multitude of tasks. |
Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses (D19-1)
Copied to clipboard
| Challenge: | Sentence position is a strong feature for news summarization, since the lead often summarizes the key points of the article. |
| Approach: | They propose two techniques to make neural systems sensitive to the importance of content in different parts of the article by using random shuffled sentences to pretrain the model. |
| Outcome: | The proposed techniques improve the performance of a competitive reinforcement learning based extractive system, with the auxiliary loss being more powerful than pretraining. |
Recycle Your Wav2Vec2 Codebook: A Speech Perceiver for Keyword Spotting (2022.coling-1)
Copied to clipboard
| Challenge: | Pretraining a keyword Spotting model with a pretraining encoder is expensive and requires a quadratic cost. |
| Approach: | They propose to recycle phonetic information encoded in wav2vec2.0's latent codebook, which has been typically thrown away after pretraining. |
| Outcome: | The proposed model can be initialized with phonetic embeddings, and it delivers accuracy gains at no latency costs. |
Simple and Effective Input Reformulations for Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Foundation language models learn from their finetuning input context in different ways. |
| Approach: | They propose three different data efficient techniques to improve translation performance . they reformulate inputs during finetuning for challenging translation tasks . |
| Outcome: | The proposed techniques show significant improvements on the Flores200 translation benchmark. |
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends (2026.findings-acl)
Copied to clipboard
Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Qiang Sun, Geoffrey Martin, Wei Liu, Yifan Peng
| Challenge: | Visually Rich Document Understanding (VRDU) frameworks are a key area of research . early approaches to VRDU relied on manually crafted rules and domain-specific heuristics . conventional deep learning approaches do not integrate the diverse modalities in documents . |
| Approach: | They review recent advances in MLLM-based Visually Rich Document Understanding (VRDU) their findings highlight emerging trends and promising research directions . |
| Outcome: | The proposed frameworks are scalable, reliable, and adaptable, the authors argue . their findings highlight emerging trends and promising research directions . |
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)
Copied to clipboard
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina
| Challenge: | Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance . |
| Approach: | They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. |
| Outcome: | The proposed model can be extended to other languages without incurring prohibitively large costs. |
On the Calibration of Large Language Models and Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are becoming more popular and are proving to be reliable . however, their reliability is often understudied due to their uncertainty and complex structure . |
| Approach: | They conduct a systematic examination of the calibration of aligned language models throughout the entire construction process including pretraining and alignment training. |
| Outcome: | The results shed light on whether popular large language models are well-calibrated and how the training process influences model calibration. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
Bridging the Memorization-Utilization Gap: Near-Lossless Context Compression via Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in context compression have failed to effectively utilize compressed representations for downstream tasks. |
| Approach: | They propose a holistic training paradigm that uses outcome-based RL to enable implicit expansion. |
| Outcome: | The proposed model outperforms previous models on NIAH, LongBench and multi-hop reasoning. |
Downstream Datasets Make Surprisingly Good Pretraining Corpora (2023.acl-long)
Copied to clipboard
| Challenge: | a dominant practice is to fine tune large pretrained transformer models using smaller downstream datasets . performance gains are not always attributable to the use of external data in massive amounts . |
| Approach: | They propose to use the same (downstream) training data for pretraining and finetuning to compare models. |
| Outcome: | The proposed model outperforms standard pretraining on the BookWiki corpus on 7 and 5 datasets. |
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation (2021.emnlp-main)
Copied to clipboard
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, Pascale Fung
| Challenge: | Lack of publicly available NLG benchmarks for low-resource languages poses a challenge . authors show that IndoBART and IndoGPT achieve competitive performance on all tasks . |
| Approach: | They propose a benchmark to measure natural language generation progress in three low-resource languages of Indonesia . they use a corpus of pretraining datasets to build their models . |
| Outcome: | The proposed benchmark measures progress in Indonesian, Javanese, and Sundanese . the results highlight the importance of pretraining on closely related, localized languages . |
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020.emnlp-main)
Copied to clipboard
| Challenge: | a representative pretraining model is fit to a diverse YouTube8M dataset . a priori, this domain is relatively easy for instructional videos . |
| Approach: | They fit a representative pretraining model to a YouTube8M dataset and examine its success and failure cases. |
| Outcome: | The proposed model can be trained on more diverse video corpora and achieve high performance on many video understanding tasks. |
Craw4LLM: Efficient Web Crawling for LLM Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work discards over 90% of the raw data collected from web crawls, highlighting the inefficiency of current web crawlers in collecting LLM pretraining data. |
| Approach: | They propose a web crawling method that leverages the preference of LLMs as the priority score of the web crawler’s scheduler to obtain high-quality pretraining data. |
| Outcome: | The proposed method achieves high-quality pretraining data on a web graph containing 900 million webpages from a commercial search engine's index with just 21% URLs crawled. |
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream task instances as a language modeling problem. |
| Approach: | They propose to reformulate downstream tasks as a language modeling problem and add a regularization that preserves pretraining weights to the model to mitigate the destructive tendency of few-shot finetuning. |
| Outcome: | The proposed model performs better on low data regimes than the standard model on few-shot finetuning. |
Context versus Prior Knowledge in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have investigated how often a model will rely on prior knowledge over conflicting contextual information in answering questions. |
| Approach: | They propose two mutual information-based metrics to measure a model’s dependency on a context and on its prior about an entity. |
| Outcome: | The proposed metrics show that language models can integrate prior knowledge and new information in a predictable way across different questions and contexts. |
Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge (2025.findings-acl)
Copied to clipboard
| Challenge: | Data watermarking in language models injects traceable signals, such as specific token sequences or stylistic patterns, into copyrighted text, allowing copyright holders to track and verify training data ownership. |
| Approach: | They propose a data watermarking approach that injects coherent and plausible yet fictitious knowledge into training data using generated passages describing a fictious entity and its associated attributes. |
| Outcome: | The proposed method is designed to be memorized by the LLM, and that increasing their density, length, and diversity of attributes strengthens their memorization. |
Continual Contrastive Finetuning Improves Low-Resource Relation Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Relation extraction (RE) has been challenging in low-resource domains and with limited resources. |
| Approach: | They propose to pretrain and finetune the RE model using consistent objectives of contrastive learning. |
| Outcome: | The proposed method outperforms PLM-based RE classifier on two document-level RE datasets. |
Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that target a single language are not seen during finetuning, but are able to respond in multiple languages once deployed in downstream applications. |
| Approach: | They investigate the minimal amount of multilinguality required during finetuning to elicit effective cross-lingual generalisation in English-centric LLMs. |
| Outcome: | The proposed model can respond in as few as two to three languages to a user's query in English, but the degree to which a target language is seen during pretraining is limiting. |
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2021.emnlp-main)
Copied to clipboard
| Challenge: | Massively multilingual language models offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks, but there is a profound performance gap between resource-rich and resource-poor target languages. |
| Approach: | They propose a series of data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to low-resource languages and unseen scripts. |
| Outcome: | The proposed methods improve learning of the new dedicated embedding matrix in the target language and for low-resource languages written in unseen scripts. |
PISCO: Pretty Simple Compression for Retrieval-Augmented Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Document compression methods suffer from accuracy losses and limited context size. |
| Approach: | They propose a method that achieves a 16x compression rate with minimal accuracy loss . they show that PISCO outperforms existing compression models by 8% in accuracy . |
| Outcome: | The proposed method outperforms existing compression models by 8% in accuracy. |
Representational Isomorphism and Alignment of Multilingual Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing isomorphism of sentence representations can facilitate representational alignments in zero-shot and few-shot settings. |
| Approach: | They propose to apply a contrastive objective to LLMs with a small number of translation pairs to improve models' performance on Semantic Textual Similarity tasks. |
| Outcome: | The proposed representation-level approach significantly improves on Semantic Textual Similarity (STS) tasks across languages even without a monolingual objective. |
Model-Based Imaginative Planning for Embodied Agents (2026.acl-long)
Copied to clipboard
Junru Song, Hengzhe Jin, Yucong Huang, Tingsong Jiang, Weien Zhou, Feifei Wang, Yang Yang, Ying Wen, Wen Yao
| Challenge: | a lightweight world model converts raw pixels into object-centric symbolic states amenable to language-based reasoning . IMPLEMENT is a framework for grounding language agents in visual embodied environments . |
| Approach: | They propose a model-based reasoning framework that enables frozen large language models to perform imaginative planning. |
| Outcome: | The proposed framework can be used to ground language agents in visual embodied environments. |
IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization (2021.emnlp-main)
Copied to clipboard
| Challenge: | In IndoBERTweet, a pretraining model for Indonesian Twitter is extended with domain-specific vocabulary. |
| Approach: | They propose a pretraining model that extends a monolingual Indonesian BERT model with domain-specific vocabulary. |
| Outcome: | The proposed model can be initialized with the average BERT subword embedding five times faster than existing methods for vocabulary adaptation. |
GRASS: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing projection-based methods that project gradients into a lower-dimensional subspace can introduce computational and memory overheads. |
| Approach: | They propose a novel approach that leverages sparse projections to transform gradients into structured sparser updates. |
| Outcome: | The proposed approach significantly reduces memory usage for optimizer states and minimizes memory footprint, computation, and communication costs, leading to substantial throughput improvements. |
Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards (2026.acl-long)
Copied to clipboard
Youliang Yuan, Qiuyang Mang, Jingbang Chen, Hong Wan, Xiaoyuan Liu, Junjielong Xu, Jen-tse Huang, Wenxuan Wang, Wenxiang Jiao, Pinjia He
| Challenge: | Existing models are susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability. |
| Approach: | They propose a Rubric Reward Model that rewards the entire reasoning trajectory against problem-specific rubrics. |
| Outcome: | The proposed model outperforms outcome-only supervision on four math benchmarks and boosts Verified Pass@1024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%. |
Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge (2023.acl-long)
Copied to clipboard
| Challenge: | CR is the ability to understand and navigate the world using basic knowledge and understanding shared by most people. |
| Approach: | They propose to incorporate pretrained knowledge into NMT models and use them as robust testbeds for investigating CR in NMT. |
| Outcome: | The proposed method improves the training of NMT models with high CR abilities and provides accurate evaluation metrics. |
Mitigating Temporal Misalignment by Discarding Outdated Facts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal misalignment is a problem for knowledge-intensive tasks where models must rely on data from the past to make predictions. |
| Approach: | They propose a temporal misalignment task to predict how long a given fact will remain true. |
| Outcome: | The proposed task improves calibration for knowledge-intensive tasks under temporal misalignment by discarding volatile facts. |
A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. |
| Approach: | They propose to compare multilingual pretraining objectives in a controlled methodological environment with multilingual models. |
| Outcome: | The proposed model outperforms existing models in 6 languages and demonstrates that multilingual translation is an effective pretraining objective under the right conditions. |
Set the Clock: Temporal Alignment of Pretrained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. |
| Approach: | They construct a time-sensitive question dataset and use it to examine temporal alignment methods to align their internal knowledge to a target time. |
| Outcome: | The proposed methods improve LLaMa2's performance by 62% if they are fine tuned to the year 2022 . |
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024.findings-acl)
Copied to clipboard
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang
| Challenge: | Recent vision-language models (VLMs) have shown impressive capabilities as general visual assistants, but there are two challenges to their performance: (1) lacking task diversity in pretraining and visual instruction tuning; (2) annotation error and bias in GPT-4 synthesized instruction tuning data. |
| Approach: | They propose a two-stage instruction tuning framework that fine tunes VLMs firstly and further tuned on GPT-4 synthesized data. |
| Outcome: | The proposed framework outperforms the traditional single-stage visual instruction tuning framework and achieves state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. |
Don’t Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models (2026.acl-long)
Copied to clipboard
| Challenge: | Small language models struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters. |
| Approach: | They propose a training-free method that leverages peakedness to align schemas with pretraining knowledge to rename tool components. |
| Outcome: | Experiments on MetaTool and RoTBench show that PA-Tool significantly improves tool-use accuracy without retraining. |
Fisher-Driven Adaptive Locating for Knowledge Editing in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for locating and editing static knowledge are costly and risk catastrophic forgetting or error. |
| Approach: | They propose a Fisher-driven adaptation-aware locating strategy that dynamically identifies which model components should be edited for a given knowledge update. |
| Outcome: | Experiments on standard benchmarks show that FiDAL improves editing effectiveness and knowledge preservation across multiple editing methods. |
Precise In-Parameter Concept Erasure in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for erasing concepts from model parameters are either too coarse, shallow or ineffective. |
| Approach: | They propose a framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space. |
| Outcome: | The proposed framework reduces errors on the target concept to as low as 7.7% while improving specificity and robustness. |
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities, particularly in English, but for less prevalent languages, performance can be significantly lower, making additional adaptation paramount. |
| Approach: | They propose a new adaptation method based on iteratively merging multiple models fine-tuned on a subset of available training data that reduces forgetting while maintaining learning on the target domain. |
| Outcome: | The proposed method outperforms LLAMA-3-8B-based models in German and German while maintaining learning on the target domain. |
TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm. |
| Approach: | They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models. |
| Outcome: | The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks. |
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research demonstrates that multilingual encoder models are capable of zero-shot cross-lingual learning by using cloze-style prompts. |
| Approach: | They propose to reformulate input examples into cloze-style prompts to perform zero-shot multilingual tasks or linguistic probing by predicting label words at the masked token position. |
| Outcome: | The proposed method performs zero-shot multilingual tasks without updating parameters. |
How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models leverage parametric and in-context knowledge in training . however, when these sources conflict, models arbitrate based on their internal confidence . |
| Approach: | They conduct controlled experiments using synthetic corpora to identify data properties that shape knowledge utilization. |
| Outcome: | The results show that the robust use of both knowledge sources is an emergent property . the results provide guidance for designing training data that supports the reliability of parametric and in-context knowledge in language models. |
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data (2025.findings-acl)
Copied to clipboard
Xuemiao Zhang, Xu Liangyu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, Xunliang Cai
| Challenge: | Existing methods of uniformly sampling data throughout the pretraining process are suboptimal because they overlook the model's evolving data preferences. |
| Approach: | They propose a Perplexity Difference (PD) based Preference Curriculum learning framework which perceives and uses the data preferred by LLMs as their capabilities improve . they propose PDPC to complete the arrangement of the dataset offline and ensure continuous training without interruption. |
| Outcome: | The proposed framework surpasses baselines on 1.3B and 3B models and achieves an increased average accuracy of over 8.1% across MMLU and CMMLU. |
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean (2024.lrec-main)
Copied to clipboard
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
| Challenge: | Large language models (LLMs) use pretraining to predict the subsequent word, but less-resourced languages are being overlooked. |
| Approach: | They propose to expand the MLLM vocabularies to enhance expressiveness and use bilingual data for pretraining to align the high- and less-resourced languages. |
| Outcome: | The proposed model outperforms existing models in qualitative analyses compared to Korean monolingual models. |
Distinguishing fair from unfair compositional generalization tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Compositional generalization benchmarks assess learning agents' ability to combine familiar concepts in novel ways. |
| Approach: | They propose to use compositional generalization benchmarks to assess learning agents' ability to combine familiar concepts in novel ways. |
| Outcome: | The proposed tasks are easy and hard, but no present-day models get any traction. |
Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLMs (2025.findings-emnlp)
Copied to clipboard
Chenkun Tan, Pengyu Wang, Shaojun Zhou, Botian Jiang, Zhaowei Li, Dong Zhang, Xinghao Wang, Yaqian Zhou, Xipeng Qiu
| Challenge: | Recent advances in multimodal large language models focus on improving performance . however, language prior conflict leads to suboptimal vision-language alignment . |
| Approach: | They propose a method to decouple the alignment process from language prior interference . they use a proxy LLM to detach from language interference during pretraining . |
| Outcome: | The proposed method improves training performance and generalizes training data. |
Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Fig. 1 and 2 shows that log-likelihood vectors provide a consistent representation for language models . weight permutation symmetries and architectural dependencies hinder direct comparisons between models with different learning methods or designs. |
| Approach: | They propose a log-likelihood vector for comparing language models as probability distributions . they establish a consistent scale for KL divergence across various settings . |
| Outcome: | The proposed model comparisons show that the log-likelihood space is smaller than the weight space . the proposed model compares language models across checkpoints, model sizes, quantization, fine-tuning, and layers . |
CoEx – Co-evolving World-model and Exploration (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LLM agents fail to assimilate new observations into dynamic updates of the world model, leading to divergent and erroneous plans. |
| Approach: | They propose a hierarchical agent architecture that allows LLM planning to co-evolve with a dynamically updated model of the world. |
| Outcome: | The proposed agent outperforms existing agent paradigms in planning and exploration. |
CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) achieve excellent performance through pretraining on extensive data. |
| Approach: | They propose an efficient selective layer intervention based on parameter-efficient fine-tuning methods to select the optimal steering layer to modulate LLM semantics. |
| Outcome: | The proposed approach is based on a model-agnostic framework and is safe to deploy. |
MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context. |
| Approach: | They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans. |
| Outcome: | The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases. |
Improving Preference Alignment of LLM with Inference-Free Self-Refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Approach: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Outcome: | Experiments show that incorporating IFSR into preference alignment yields performance improvement over 10%. |
How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance. |
| Approach: | They propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
| Outcome: | The proposed model predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
The Effects of Pretraining in Video-Guided Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to improve VMT models integrate text and video modalities. |
| Approach: | They propose an approach that improves the performance of VMT models by using a new dataset which contains transcribed audio descriptions of movies. |
| Outcome: | The proposed model improves on the MAD (Movie Audio Descriptions) dataset. |
Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning. |
| Approach: | They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations. |
| Outcome: | The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs. |
To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)
Copied to clipboard
| Challenge: | State-of-the-art language models perform well on a variety of language tasks, but struggle with understanding negation cues in tasks like natural language inference (NLI). |
| Approach: | They propose a new learning strategy for negation building on ELECTRA’s replaced token detection objective. |
| Outcome: | The proposed approach leads to substantial gains on a variant of RTE with additional negation. |
MAviS: A Multimodal Conversational Assistant For Avian Species (2025.emnlp-main)
Copied to clipboard
Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jinxing Zhou, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal
| Challenge: | Existing multimodal large language models face challenges when it comes to specialized topics like avian species. |
| Approach: | They propose a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species. |
| Outcome: | The proposed model outperforms the baseline MiniCPM-o-2.6 by a large margin. |
Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to training large language models suffer from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. |
| Approach: | They propose a framework that integrates process supervision into group relative policy optimization. |
| Outcome: | The proposed framework outperforms standard GRPO on knowledge-intensive benchmarks by 5.0% and 6.3% on Qwen3-1.7B. |
HORIZON: A Benchmark for In-the-wild User Behaviour Modeling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing user modeling benchmarks focus on short sessions and next-item prediction within a single domain. |
| Approach: | They propose a benchmark that reformulates user modeling along three axes . it covers 54M users and 35M items, enabling pretraining and evaluation . they propose tasks and evaluation setups that better reflect real-world deployment scenarios . |
| Outcome: | The proposed benchmark covers 54M users and 35M items, and is based on Amazon Reviews. |
Multilingual Pretraining for Pixel Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL-M4 model pretrains on four visually and linguistically diverse languages . previous work on pixel-based language models focused on monolingual pretraining on English data . |
| Approach: | They propose a pixel-based language model that is pretrained on four visually diverse languages. |
| Outcome: | The proposed model outperforms an English-only counterpart on non-Latin scripts on semantic and syntactic tasks. |
Rethinking the Idiomaticity Decomposability Hypothesis: Evidence from Distributional Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Decomposability is thought to predict syntactic flexibility, but is not attributed to distributional experience. |
| Approach: | They propose a model-internal measure of decomposability and relate it to human ratings, syntactic flexibility, and predictability while tracking idiom learning during pretraining. |
| Outcome: | The proposed model-internal measure correlates weakly with human judgments and shows a small but consistent negative relationship with syntactic flexibility. |
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models often encode word-form variation as linear directions in the embedding space. |
| Approach: | They propose a compact reshaping of large language models' vocabulary by using shared vectors instead of unique tokens. |
| Outcome: | The proposed approach frees 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. |
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | a new approach to adapt generalist models to expert domains is needed to overcome this problem. |
| Approach: | They propose a parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. |
| Outcome: | The proposed approach reduces training time by 35-55% over continual pretraining and reduces parameter counts up to 37% w.r.t expansion-only methods. |
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments (2026.acl-long)
Copied to clipboard
Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Marinas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas John Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clément Charmillot, Jonathan Coles, Jan Milan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Miserlis Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Ioannis Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javier Rando, Mathieu Sauser, Jakhongir Saydaliev, Mukhammadali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas C. Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag
| Challenge: | Apertus is a fully open suite of large language models (LLMs) designed to address responsibility shortcomings in today’s open model ecosystem, namely data responsibility and global representation. |
| Approach: | They propose to release a fully open suite of large language models (LLMs) that address data responsibility and global representation shortcomings in today’s open model ecosystem. |
| Outcome: | The proposed model is pretrained on openly available data and suppresses verbatim recall of data while retaining task performance. |