DOWNSTREAM

Papers with downstream

Tokenization Is More Than Compression (2024.emnlp-main)

Copied to clipboard

Craig Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, Chris Tanner

Challenge:	Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens.
Approach:	They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance.
Outcome:	The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance.

Counterfactual Augmentation for Multimodal Learning Under Presentation Bias (2023.findings-emnlp)

Copied to clipboard

Victoria Lin, Louis-Philippe Morency, Dimitrios Dimitriadis, Srinagesh Sharma

Multimodality

Challenge:	In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage.
Approach:	They propose a method for correcting presentation bias using generated counterfactual labels by augmentation of the labels by the user.
Outcome:	The proposed method improves performance in an oracle setting compared to uncorrected models and existing bias-correction methods.

Modeling Information Change in Science Communication with Semantically Matched Paraphrases (2022.emnlp-main)

Copied to clipboard

Dustin Wright, Jiaxin Pei, David Jurgens, Isabelle Augenstein

Text Generation

Challenge:	Whether the media faithfully communicate scientific information has long been a core issue to the science community.
Approach:	They propose to use the SCIENTIFIC PARAPHRASE AND INFORMATION CHANGE DATASET to identify paraphrased scientific findings annotated for degree of information change to enable large-scale tracking and analysis of information changes in science communication.
Outcome:	The proposed dataset contains 6,000 scientific finding pairs extracted from news stories, social media discussions, and full texts of original papers.

Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER (2025.emnlp-industry)

Copied to clipboard

Junyi Zhu, Savas Ozkan, Andrea Maracani, Sinan Mutlu, Cho Jung Min, Mete Ozay

Information Extraction & Text Mining

Challenge:	nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance.
Approach:	They propose a framework that enables a single shared encoder backbone with modular adapters.
Outcome:	The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint.

Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)

Copied to clipboard

Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, George Karypis

Semantic Text Processing

Challenge:	Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.
Approach:	They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks.
Outcome:	The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task.

Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)

Copied to clipboard

Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, Nikolaos Aletras

Language Models

Challenge:	Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations.
Approach:	They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective.
Outcome:	Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture.

The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023.findings-emnlp)

Copied to clipboard

Mutian He, Philip Garner

Multilinguality

Challenge:	Modern artificial intelligence is characterized by large pretrained language models with strong language capabilities to be adapted to various downstream tasks.
Approach:	They propose to use the task of speech translation (ST) to pretrain speech models for end-to-end SLU on intra- and cross-lingual scenarios.
Outcome:	The proposed model achieves higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks.

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022.naacl-main)

Copied to clipboard

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren

Language Models

Challenge:	Pretrained language models are typically learned over a large, static corpus and fine-tuned for various downstream tasks.
Approach:	They propose to continuously update a pretrained language model to adapt to emerging data and to keep track of the model's performance.
Outcome:	The proposed model can adapt to new corpora while retaining knowledge in earlier domains.

A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank (2022.naacl-main)

Copied to clipboard

Dan Malkin, Tomasz Limisiewicz, Gabriel Stanovsky

Language Models

Challenge:	Pretraining languages improve cross-lingual transfer for BERT-based models . Interestingly, PLMs exhibit zero-shot cross-linguistic abilities on downstream examples in languages seen only during pretraining.
Approach:	They develop a quadratic time complexity method to estimate pretraining languages' relations between linguistic features and two downstream tasks.
Outcome:	The proposed method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks.

An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference (2024.findings-emnlp)

Copied to clipboard

Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras

Language Models

Challenge:	Cross-lingual vocabulary adaptation (CVA) methods have been proposed for adapting models to a target language . but effectiveness of these methods on increasing inference efficiency of generative large language models has not been explored.
Approach:	They propose to use cross-lingual vocabulary adaptation methods to adapt models to a target language to improve downstream performance.
Outcome:	The proposed methods significantly speed up models in four languages and four natural language understanding tasks.

Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets (2023.emnlp-main)

Copied to clipboard

Irina Bejan, Artem Sokolov, Katja Filippova

Challenge:	Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP, however, data quality might have already become the bottleneck to unlock further gains.
Approach:	They propose a general method for improving model performance in the presence of noisy training data based on self-influence and bandit curriculum learning.
Outcome:	The proposed method improves model performance in machine translation, question answering and text classification, building up on approaches to self-influence calculation and automated curriculum learning.

Simple and Effective Input Reformulations for Translation (2023.emnlp-main)

Copied to clipboard

Brian Yu, Hansen Lillemark, Kurt Keutzer

Multilinguality

Challenge:	Foundation language models learn from their finetuning input context in different ways.
Approach:	They propose three different data efficient techniques to improve translation performance . they reformulate inputs during finetuning for challenging translation tasks .
Outcome:	The proposed techniques show significant improvements on the Flores200 translation benchmark.

Weak2Wise: An Automated, Lightweight Framework for Weak-LLM-Friendly Reasoning Synthesis (2025.findings-emnlp)

Copied to clipboard

Jianing Lin, Yuanfang Guo, Shunning Liu, Zeming Liu, Yunhong Wang

Reasoning

Challenge:	Existing approaches to finetuning large language models rely on expensive manual annotations or auxiliary models and fail to address the unique constraints of smaller "weak" LLMs.
Approach:	Weak2Wise is a fully automated framework for synthesizing highquality, weak-LLM-friendly reasoning traces.
Outcome:	Weak2Wise is a fully automated, lightweight framework for synthesizing highquality, weak-LLM-friendly reasoning traces.

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models (2025.emnlp-main)

Copied to clipboard

Kangtao Lv, Haibin Chen, Yujin Yuan, Langming Liu, Shilei Liu, Yongwei Wang, Wenbo Su, Bo Zheng

Language Models

Challenge:	Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance.
Approach:	They propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts.
Outcome:	The proposed model predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts.

Text Filtering Classifiers for Medium-Resource Languages (2024.lrec-main)

Copied to clipboard

Jón Daðason, Hrafn Loftsson

Information Retrieval

Challenge:	linguistic and NLP researchers use web-crawled corpora to filter low-quality texts . early Transformer-based language models were typically pre-trained on curated corporum .
Approach:	They compare the effectiveness of various text filtering classifiers on Icelandic, Estonian and Basque texts . they use a perplexity-based classifier and a self-supervised classifier trained on TQ-IS to discern between documents from curated and web-crawled corpora.
Outcome:	The proposed classifiers achieve F1 scores of 94.48%, 99.01% and 93.40% on the Icelandic, Estonian and Basque datasets.

Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models (2026.findings-acl)

Copied to clipboard

Kainan Liu, Yong Zhang, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao

Semantic Text Processing

Challenge:	Existing methods for fine-tuning pre-trained models are limited due to suboptimal activation subspaces.
Approach:	They propose a method that leverages tail eigenvectors of model output activations to construct low-rank adapters.
Outcome:	The proposed method outperforms existing methods across 16 benchmarks and surpasses full fine-tuning in certain scenarios.

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models (2025.emnlp-main)

Copied to clipboard

Tomohiro Sawada, Kartik Goyal

Language Models

Challenge:	Existing work shows that byte-pair encoding (BPE) tokenization uses a learned merge list to iteratively combine subword units into tokens during inference time.
Approach:	They propose to use a standard byte-pair encoding algorithm to pair a learned token vocabulary with a detailed merge list to compress text.
Outcome:	The proposed algorithms differ from the encoding process during training and show that the targetted deviation from merge lists exhibits significant degradation in language model performance.