Papers with Fine-tuning
SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent research shows promising results on combining pretrained language models with canonical utterance for few-shot semantic parsing. |
| Approach: | They propose a few-shot semantic parsing method that decomposes a problem into a sequence of sub-problems, which correspond to the sub-clauses of the formal language. |
| Outcome: | The proposed method achieves SOTA performance of BART-based models on GeoQuery and EcommerceQuery, which are two few-shot datasets with compositional data split. |
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on instruction learning, where a model learns to perform unseen tasks from task descriptions alone. |
| Approach: | They propose to use a controlled synthetic environment to characterize large transformer models as instruction learners. |
| Outcome: | The proposed model can interpret only 65.6% of test instructions and 11%-24% of instructions in out-of-distribution settings. |
MedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries (2023.acl-srw)
Copied to clipboard
| Challenge: | Clinical texts contain important temporal information, such as medication start and end dates, appointment dates, and diagnosis dates. |
| Approach: | They propose to use prompt-based learning and fine-tuning to classify temporal relations between treatments and hospitalisation periods in discharge summaries. |
| Outcome: | The proposed method identifies whether a treatment was administered between the time of admission and discharge from the hospital. |
MoFE: Mixture of Frozen Experts Architecture (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are characterized by their immense size, often consisting of at least one billion parameters. |
| Approach: | They propose a mixture of Frozen Experts architecture that integrates PEFT and MoE to enhance both training efficiency and model scalability. |
| Outcome: | The proposed architecture outperforms other methods while achieving the highest efficiency. |
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought (2026.findings-eacl)
Copied to clipboard
| Challenge: | Lack of perceptual grounding limits vision-language models' ability to interpret visual data . prior work on visualized data understanding focused on adapting VLMs to instruction tuning and chain-of-thought supervision . |
| Approach: | They propose a framework that enhances visual reasoning through human-like interpretation grounding. |
| Outcome: | The proposed framework improves on ChartQA and ChartQAPro benchmarks by +11.2%. |
Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure (2026.eacl-srw)
Copied to clipboard
| Challenge: | a sparsity-exploiting backward pass is a memory-efficient way to accelerate LLM fine-tuning. |
| Approach: | They propose a method that exploits padding-induced gradient sparsity to accelerate backward computation. |
| Outcome: | The proposed method achieves a backward pass speedup of 2.15x on GLUE and 1.99x on reasoning benchmarks while maintaining memory usage identical to the regular PyTorch fine-tuning. |
Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer (2020.aacl-main)
Copied to clipboard
| Challenge: | Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. |
| Approach: | They propose to combine embeddings from a pre-trained model with a simple sentence embeddable model. |
| Outcome: | The proposed approach outperforms FT on small datasets with negligible computational overhead. |
RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have strong generalizability, but fine-tuning involves updating all parameters, rendering full fine-uning resource-intensive. |
| Approach: | They propose a parameter-efficient fine-tuning method that updates all pre-trained parameters during inference. |
| Outcome: | The proposed method outperforms baseline methods on five benchmarks across 20 datasets. |
Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive. |
| Approach: | They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer. |
| Outcome: | The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment. |
FPC: Fine-tuning with Prompt Curriculum for Relation Extraction (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction ignore semantics of relation labels . prompt-based fine-tuning has been proposed for RE . |
| Approach: | They propose a method for relation extraction using prompt-based fine-tuning . they use auxiliary prompt-tuned learning task to make the model capture semantics of relation labels . |
| Outcome: | The proposed method outperforms existing methods on four widely used RE benchmarks under fully supervised and low-resource settings. |
Culture Cartography: Mapping the Landscape of Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can empower users to be more knowledgeable, productive, and creative, but their utility is often diminished for under-represented groups and cultures. |
| Approach: | They propose a methodology that operationalizes a mixed-initiative approach to finding culture-specific knowledge that is salient to in-group users but unknown to LLMs. |
| Outcome: | The proposed method improves the accuracy of LLMs on culturally-competent language models by 19.2%. |
The Multilingual Microblog Translation Corpus: Improving and Evaluating Translation of User-Generated Text (2022.lrec-1)
Copied to clipboard
| Challenge: | a corpus of over 200,000 microblog translations supports translation of thirteen languages into English . large collections of parallel text, or bitext, are increasingly available in many languages . |
| Approach: | They propose a corpus of over 200,000 microblog posts that supports translation of thirteen languages into English. |
| Outcome: | The proposed corpus contains over 200,000 translations of microblog posts in 13 languages . fine-tuning showed significant improvements in translation quality . |
Feature Drift: How Fine-Tuning Repurposes Representations in LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are a powerful tool for interpreting neural networks by extracting concepts (features) represented in their activations. |
| Approach: | They propose to use Sparse Autoencoders to extract concepts from their activations to explain how fine-tuning changes model capabilities. |
| Outcome: | The proposed model recombines existing concepts rather than learning new ones, and shows that it is a better explanation for how fine-tuning changes model capabilities. |
Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similarity (2024.findings-eacl)
Copied to clipboard
Eric Khiu, Hasti Toossi, Jinyu Liu, Jiaxu Li, David Anugraha, Juan Flores, Leandro Roman, A. Seza Doğruöz, En-Shiun Lee
| Challenge: | Existing approaches for predicting the performance of NLP models for low-resource languages (LRLs) focus on high-resourced languages, overlooking LRLs and domain shifts. |
| Approach: | They investigate the impact of domain similarity on predicting performance of machine translation models in low-resource languages. |
| Outcome: | The results show that domain similarity has the most important impact on predicting the performance of Machine Translation models. |
Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the detection and aggregation of media bias lack a gold standard data set and high context dependencies. |
| Approach: | They propose to use a data set to identify media bias by word and sentence level . they propose to train a model to detect bias-inducing sentences in news articles automatically . |
| Outcome: | The proposed model outperforms existing methods on a large corpus of labels on the word and sentence level. |
AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories (2024.findings-emnlp)
Copied to clipboard
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li
| Challenge: | Existing studies focus on specialized agents designed for particular tasks. |
| Approach: | They propose to scale annotated interaction trajectories and fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. |
| Outcome: | The proposed model can scale to get generalized agent capabilities. |
Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches focus on likelihood-based training or using reinforcement learning to fine-tune models based on a single reward. |
| Approach: | They propose an approach to fine-tune programs from natural language instruction . they propose a reward function that linearly combines them and a policy for program generation . |
| Outcome: | The proposed approach achieves better performance than competing methods using Reinforcement Learning. |
Index-Time Prefix Injection for Multi-Tenant Retrieval: Improving Search Relevance Without Model Fine-Tuning (2026.acl-industry)
Copied to clipboard
| Challenge: | a single multilingual biencoder handles all retrieval, but these are task-generic and domain-agnostic. |
| Approach: | They propose a training-free method that prepending domain-descriptive prefixes to documents during indexing. |
| Outcome: | The proposed method improves retrieval relevance by prepending natural-language prefixes to documents during indexing. |
T-VEC: A Telecom-Specific Vectorization Model with Enhanced Semantic Understanding via Deep Triplet Loss Fine-Tuning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Generic embedding models struggle to represent telecom-specific semantics . specialized terminology and ambiguous terms often limit their utility in retrieval and downstream tasks. |
| Approach: | They propose a domain-adapted embedding model fine-tuned from a gte-Qwen2-1.5B-instruct backbone. |
| Outcome: | The proposed model outperforms MPNet, BGE, Jina and E5 on a custom benchmark . it is open source and has a triplet loss objective . |
Self-Supervised Position Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for debiasing large language models require external bias knowledge or annotated non-biased samples, which is lacking for position debiases. |
| Approach: | They propose a self-supervised position debiasing framework that leverages unsupervised responses from pre-trained LLMs for debiazing without external bias knowledge. |
| Outcome: | The proposed framework outperforms existing methods in mitigating three types of position biases on eight datasets and five tasks. |
An Evaluation of Progressive Neural Networksfor Transfer Learning in Natural Language Processing (2020.lrec-1)
Copied to clipboard
Abdul Moeed, Gerhard Hagerer, Sumit Dugar, Sarthak Gupta, Mainak Ghosh, Hannah Danner, Oliver Mitevski, Andreas Nawroth, Georg Groh
| Challenge: | Fine-tuning suffers from catastrophic forgetting, a problem exacerbated in natural language processing (NLP). |
| Approach: | They propose to use progressive neural networks to re-use previously learned knowledge when learning new tasks. |
| Outcome: | The proposed approach improves on common NLP tasks across a range of architectures, datasets, and tasks. |
Reliable Gradient-free and Likelihood-free Prompt Tuning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Large pre-trained language models are often offered as black-box APIs due to privacy or commercial constraints. |
| Approach: | They propose to tune the soft prompts without requiring gradient computation and extend the model to include a distribution over prompts. |
| Outcome: | The proposed methods are competitive with gradient-based approaches with full access to the PLM. |
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study validates the effectiveness of chat language models by fine-tuning instruction data. |
| Approach: | They propose to use a large-scale dataset of instructional conversations to fine-tune a conversational model on instruction data. |
| Outcome: | The proposed model outperforms open-source models in key metrics including scale, average length, diversity, coherence, etc. |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning and in-context learning are two prevalent methods in imbuing large language models with task-specific knowledge. |
| Approach: | They propose to use a circuit shift theory to explain why in-context learning is superior to fine-tuning for tasks with implicit patterns. |
| Outcome: | The proposed method can grasp deep patterns and significantly improve accuracy on implicit patterns, compared with fine-tuning and in-context learning. |
ChildTalk: A Multi-Dialect Chinese Child Speech Corpus with Full-Length Child–Caregiver Conversations for Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition (ASR) for children remains challenging due to developmental variability and the scarcity of high-quality corpora. |
| Approach: | They propose a large-scale Chinese child speech corpus that contains 112.5 hours of speech from 498 children and 500 caregivers. |
| Outcome: | The proposed model improves in-domain and cross-domain performance on children's speech. |
Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations (2024.findings-naacl)
Copied to clipboard
| Challenge: | supervised systems have not replaced dedicated supervised models for machine translation tasks. |
| Approach: | They propose to guide LLMs to post-edit MT with feedback from MQM annotations . they then fine-tune the LLM to improve its ability to exploit the feedback . |
| Outcome: | The proposed model improves TER, BLEU and COMET scores on Chinese-English, English-German and English-Russian data. |
Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevailing practice for adapting language models (LMs) to new domains. |
| Approach: | They propose a mask specific language model that weights the importance of domain-specific terms during fine-tuning to avoid insensitivity. |
| Outcome: | The proposed approach outperforms advanced masking strategies such as span- and PMI-based masking. |
Coarse-Tuning for Ad-hoc Document Retrieval Using Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing fine-tuning techniques for information retrieval systems require learning query representations and query-document relations. |
| Approach: | They propose a method that bridges pre-training and fine-tuning by learning query representations and query-document relations in coarse-tuned models. |
| Outcome: | The proposed method significantly improves MRR and/or nDCG@5 in four ad-hoc document retrieval datasets. |
EmoCharacter: Evaluating the Emotional Fidelity of Role-Playing Agents in Dialogues (2025.naacl-long)
Copied to clipboard
| Challenge: | EmoCharacter evaluates emotional fidelity of role-playing agents in dialogues . current evaluations focus on personality fidelity, tone imitation, and knowledge consistency . |
| Approach: | They propose a benchmark to assess emotional fidelity of role-playing agents in dialogues using large language models. |
| Outcome: | The proposed benchmark measures emotional fidelity of role-playing agents and the characters they portray. |
Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs. |
| Approach: | They propose a component-wise gradient norm clipping method to adjust convergence speed for different components to improve generalization performance, convergence speed, and training stability. |
| Outcome: | The proposed method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. |
Corpora Generation for Grammatical Error Correction (N19-1)
Copied to clipboard
| Challenge: | Grammatical Error Correction (GEC) is a computational task that requires large amounts of data to solve. |
| Approach: | They propose two approaches to generate large parallel datasets for GEC using publicly available Wikipedia edit histories using minimal filtration heuristics and round-trip translation through bridge languages. |
| Outcome: | The proposed methods yield similar sized parallel corpora with around 4B tokens and are far ahead of the state-of-the-art on the CoNLL ‘14 benchmark and the JFLEG task. |
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevalent paradigm for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the LM. |
| Approach: | They propose a lightweight alternative to fine-tuning for natural language generation tasks that optimizes a sequence of continuous vectors, which they call the prefix. |
| Outcome: | The proposed approach outperforms fine-tuning in the full data setting and extrapolates better to examples with topics that are unseen during training. |
KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing code-focused resources typically fail to ensure either the breadth of coverage or verifiable correctness. |
| Approach: | They propose a synthetic dataset that provides high-quality, verifiable training data for Large Language Models for coding. |
| Outcome: | The proposed dataset surpasses Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Llama-70B in performance on coding benchmarks. |
DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive. |
| Approach: | They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality. |
| Outcome: | The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters. |
ThinkPersona: Thinking with Persona Graphs for Faithful Individualized Role-Playing (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models are increasingly utilized as role-playing agents to simulate personas in interactive settings. |
| Approach: | They propose a role-playing agent trained to explicitly ground responses in individual identity. |
| Outcome: | The proposed agent can generate persona-consistent responses in long-context dialogues while maintaining general instruction-following capabilities. |
Forward Knows Efficient Backward Path: Saliency-Guided Memory-Efficient Fine-tuning of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | a number of fine-tuning approaches are available to improve performance of large language models. |
| Approach: | They propose a memory-efficient method to minimize memory associated with cached intermediate activations. |
| Outcome: | The proposed method minimizes memory associated with cached intermediate activations while preserving accuracy. |
Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models impose substantial resource usage. |
| Approach: | They propose a parameter-efficient fine-tuning method that freezes adapters early to reduce resource usage while maintaining performance. |
| Outcome: | The proposed method reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% while maintaining performance. |
Unsupervised Fine-tuning for Text Clustering (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text clustering fine-tune pre-trained models have been limited. |
| Approach: | They propose a method to fine-tune pre-trained models unsupervisedly for text clustering by learning text representations and cluster assignments using a clustering oriented loss. |
| Outcome: | The proposed model outperforms baseline methods and achieves state-of-the-art results on three text clustering datasets. |
Musical Score Understanding Benchmark: Evaluating Large Language Models’ Comprehension of Complete Musical Scores (2026.acl-long)
Copied to clipboard
Congren Dai, Yue Yang, Krinos Li, Huichi Zhou, Shijie Liang, Zhang Bo, Enyang Liu, Ge Jin, Hongran An, Haosen Zhang, Peiyuan Jing, KinHei Lee, Zhenxuan Zhang, Xiaobing Li, Maosong Sun
| Challenge: | Existing benchmarks for musical score understanding are narrow in scope, focusing on isolated fragments, short excerpts, or multiple-choice formulations, rather than supporting holistic reasoning over entire scores. |
| Approach: | They propose a benchmark for score-level musical understanding across textual and visual modalities. |
| Outcome: | The musical score understanding benchmark contains 1,800 question-answer pairs from works by Bach, Beethoven, Chopin, Debussy, and others. |
PAP2PAT: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs (2025.findings-acl)
Copied to clipboard
| Challenge: | In patents, the description constitutes more than 90% of the document on average, yet its automatic generation remains understudied. |
| Approach: | They propose a method to generate patent documents using a research paper as an invention specification. |
| Outcome: | The proposed model can generate 1.8k patent-paper pairs describing the same inventions, but it's difficult to provide the level of detail required. |
Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference (2023.acl-long)
Copied to clipboard
Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen
| Challenge: | Existing studies attribute catastrophic forgetting to fine-tuning, and they retain pre-trained knowledge indiscriminately without identifying what knowledge is transferable. |
| Approach: | They propose a unified objective for fine-tuning to retrieve the causality back from pre-trained data and use it to mitigate negative transfer while preserving knowledge. |
| Outcome: | The proposed method outperforms state-of-the-art fine-tuning methods on commonsense QA datasets and can be implemented as a plug-in module to inflate the performance of existing QA models. |
Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a contrastive learning framework is used to fine-tune pre-trained language models with unlabeled sentences or labeled sentences. |
| Approach: | They propose a method that makes hard positives from unlabeled sentences . they use a prefix attached to a model to allow for differentiable data augmentation . |
| Outcome: | The proposed method yields significant improvements over existing methods under semi-supervised and supervised settings. |
Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current evaluation practices of open domain dialogue systems are still highly dependent on human evaluation. |
| Approach: | They propose to use an annotated dataset to evaluate chatbots using large language models. |
| Outcome: | The proposed model improves over few-shot inferences on a GPT-3.5 generated dialogue dataset. |
Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is critical for reliable predictions over text . fine-tuning with pre-trained language models has been a de facto procedure . |
| Approach: | They propose to leverage pre-trained language models for OOD detection without fine-tuning on ID data. |
| Outcome: | The proposed approach outperforms the fine-tuned model under distributional shifts. |
How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of Large Language Models for clinical numerical reasoning provide limited operation-level coverage and limited robustness of numerical understanding across clinical note formats. |
| Approach: | They propose a benchmarking tool that evaluates four main types of clinical numeracy . they present longitudinal MIMIC-IV vital-sign records in three semantically equivalent representations . |
| Outcome: | The proposed benchmark evaluates four main types of clinical numeracy: value retrieval, arithmetic computation, relational comparison, and aggregation. |
Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Nearest Neighbor Machine Translation (kNN-MT) is a powerful domain adaptation tool . the reasons for its success have not been thoroughly investigated . |
| Approach: | They propose to integrate pre-trained Neural Machine Translation models with token-level retrieval . they propose to implicitly execute gradient descent on the output projection layer of NMT . |
| Outcome: | The proposed approach outperforms model fine-tuning on in-domain tests while achieving better performance on out-of-domain sets. |
Multilingual Generation in Abstractive Summarization: A Comparative Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for multilingual generation lack thorough analysis due to extensive linguistic diversity. |
| Approach: | They propose to classify multilingual generation methodologies into three categories based on their underlying modeling principles . they introduce an automatic metric to mitigate spurious correlations associated with language mixing . |
| Outcome: | The proposed model improves in high-resource, low-resourced, and zero-shot scenarios. |
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |
WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models (2026.acl-long)
Copied to clipboard
Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong
| Challenge: | Existing agentic systems are retrieval-heavy but reasoning-light . current systems lack compositional reasoning, a key component of deep research . |
| Approach: | They propose a data synthesis pipeline WebAggregator to shift agentic paradigm . they use Proactive Explorer to collect interconnected knowledge and Compositional Logic Proposer to weave knowledge into complex questions . |
| Outcome: | The proposed pipeline surpasses GPT-4.1 and matches Claude-3.7-Sonnet on GAIA, WebWalkerQA, and XBench. |
Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass. |
| Approach: | They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens. |
| Outcome: | The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost . |
STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework (2025.findings-acl)
Copied to clipboard
Wenhao Liu, Zhenyi Lu, Xinyu Hu, Jerry Zhang, Dailin Li, Jiacheng Cen, Huilin Cao, Haiteng Wang, Yuhan Li, Xie Kun, Dandan Li, Pei Zhang, Chengbo Zhang, Yuxiang Ren, Xiaohong Huang, Yan Ma
| Challenge: | Existing datasets suffer from outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. |
| Approach: | They propose a human-in-the-loop, multi-agent data generation framework that integrates reasoning-dense filters, multiagent collaboration, and human mathematicians’ evaluations to ensure the reliability and quality of the dataset. |
| Outcome: | The proposed framework improves accuracy and quality of the 2,000-synthesized datasets by integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians’ evaluations. |
AFRIDOC-MT: Document-level MT Corpus for African Languages (2025.emnlp-main)
Copied to clipboard
Jesujoba Oluwadara Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina España-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, Davis David, Shamsuddeen Hassan Muhammad, Neo Putini, David O. Ademuyiwa, Andrew Caines, Dietrich Klakow
| Challenge: | AFRIDOC-MT is a document-level multi-parallel translation dataset covering five languages . AFRITIC-MT models perform better on sentences than general-purpose LLMs . |
| Approach: | They propose a document-level multi-parallel translation dataset covering English and five African languages. |
| Outcome: | The proposed dataset covers 334 health and 271 information technology news documents . it shows that NLLB-200 achieves the best average performance among standard models . |
RePanda: Pandas-powered Tabular Verification and Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fact-checking tabular data rely on black-box models with opaque reasoning. |
| Approach: | They propose a structured fact verification approach that translates claims into executable pandas queries. |
| Outcome: | The proposed method outperforms existing methods and demonstrates strong OOD robustness. |
Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation (2026.acl-long)
Copied to clipboard
Le Chen, Nuo Xu, Winson Chen, Bin Lei, Pei-Hung Lin, Dunzhi Zhou, Rajeev Thakur, Caiwen Ding, Ali Jannesari, Chunhua Liao
| Challenge: | Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA . |
| Approach: | They propose a dual-LLM Questioner–Solver pipeline that integrates external knowledge from compilers and runtime feedback to generate verified translations and multi-turn dialogues. |
| Outcome: | The proposed model outperforms proprietary models on key metrics like compilation success and accuracy. |
MTA:A Merge-then-Adapt Framework for Personalized Large Language Models (2026.acl-long)
Copied to clipboard
Xiaopeng Li, Yuanjin Zheng, Wanyu Wang, Wenlin Zhang, Pengyue Jia, Yingyi Zhang, Haiying He, Mengyang Ma, Yiqi Wang, Maolin Wang, Xuetao Wei, Xiangyu Zhao
| Challenge: | Personalized Large Language Models (PLLMs) aim to align outputs with individual user preferences . current methods of fine-tuning a separate module for each user are unscalable . |
| Approach: | They propose a Merge-then-Adapt framework for Personalized Large Language Models . they construct a shared Meta-LoRA bank and propose an Adaptive LoRA Fusion stage . |
| Outcome: | The proposed framework outperforms existing SOTA methods on the LaMP benchmark. |