Test-Time Self-Adaptive Small Language Models for Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent instruction-finetuned large language models (LMs) have shown notable performances in various tasks, such as question-answering. |
| Approach: | They propose to use unlabeled test data to transfer smaller language models with limited knowledge. |
| Outcome: | The proposed strategy shows significant performance improvements on benchmark QA datasets with higher robustness across diverse prompts, enabling LMs to stay stable. |
Similar Papers
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) generate factually incorrect answers based on their parametric memory. |
| Approach: | They propose a retrieval-augmented large language model that can dynamically select the most suitable strategy based on query complexity. |
| Outcome: | The proposed approach improves the performance of QA systems on open-domain QA datasets. |
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters. |
| Approach: | They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set. |
| Outcome: | The proposed method improves performance in all cases, while using a domain expert 23x smaller. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
GSM-Noise: Exploring and Enhancing Large Language Models’ Reasoning under Noisy Inputs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models struggle when dealing with complex, ill-formed, or noisy inputs . open-source models are less robust, while closed-source ones are more robust . |
| Approach: | They propose to use GSM-Noise to refine inputs before engaging in in-depth analysis to improve LLM robustness under noisy conditions. |
| Outcome: | The proposed model can achieve consistent performance gains under noisy conditions with prompt engineering, supervised finetuning, and reinforcement learning. |
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning. |
| Approach: | They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored. |
| Outcome: | The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation approaches for large language models (LLMs) rely on existing tasks and benchmarks, raising concerns about test set contamination and the genuine comprehension abilities of LLMs. |
| Approach: | They propose to evaluate LLMs by designing new tasks, automatically generating evaluation datasets for the tasks, and conducting detailed error analyses to scrutinize LLM's adaptability to new tasks. |
| Outcome: | The proposed method examines LLMs’ adaptability to new tasks, their sensitivity to prompt variations, and their error tendencies. |
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation (2022.tacl-1)
Copied to clipboard
| Challenge: | Large pretrained language models are often domain- or task-adapted via finetuning or prompting. |
| Approach: | They propose to use domain-adaptive pretraining to prepare large pretrained language models for domain- or task-adaptation by learning to learn the difference between general and adapted PLMs. |
| Outcome: | Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over finetuning or preparation via domain-adaptive pretraining. |
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving the correctness of output from large language models generate a constant number of samples per question, but Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%. |
| Approach: | They propose a model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. |
| Outcome: | The proposed technique reduces sample budget by 7.9 times with an average accuracy drop of less than 0.1%. |
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, Andre Martins
| Challenge: | Large language models (LLMs) are a promising avenue for machine translation (MT) however, their effectiveness depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. |
| Approach: | They propose a method that incorporates few-shot examples during finetuning to improve performance on MT tasks. |
| Outcome: | The proposed method outperforms few-shot prompting while eliminating the need for in-context examples. |