Influence Scores at Scale for Efficient Language Data Sampling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that ML models can be fine-tuned on as much data as possible without degradation in performance metrics. |
| Approach: | They evaluate the applicability of influence scores in language classification tasks by random sampling and stress-testing one of the scores. |
| Outcome: | The proposed model can be fine-tuned on 50% of the original data without degradation in performance metrics. |
Similar Papers
Unlearning Traces the Influential Training Data of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable abilities without being explicitly trained for such tasks. |
| Approach: | They propose a method that unlearns a test dataset and evaluates the unlearned model on training datasets. |
| Outcome: | The proposed method resembles UnTrac, while being efficient for massive training datasets. |
Do Influence Functions Work on Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Influence functions are important for quantifying the impact of individual training data points on a model’s predictions. |
| Approach: | They conduct a systematic study to address a key question: do influence functions work on large language models? |
| Outcome: | The influence functions perform poorly across multiple tasks and are therefore unsuitable for large language models. |
Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP, however, data quality might have already become the bottleneck to unlock further gains. |
| Approach: | They propose a general method for improving model performance in the presence of noisy training data based on self-influence and bandit curriculum learning. |
| Outcome: | The proposed method improves model performance in machine translation, question answering and text classification, building up on approaches to self-influence calculation and automated curriculum learning. |
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Influence-based methods show promise in achieving (1), but often struggle with (2) . data selection is often biased towards high-influence tasks, harming performance on them . |
| Approach: | They propose a Balanced and Influential Data Selection algorithm that normalizes influence scores of training data and iteratively chooses the training example with the highest influence on the most underrepresented task. |
| Outcome: | The proposed model outperforms both state-of-the-art influence-based methods and non-influence-based frameworks on seven benchmarks spanning five diverse capabilities. |
Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, prompt-based models are gaining popularity due to their easier adaptability in low-resource settings. |
| Approach: | They analyze attribution scores extracted from prompt-based models w.r.t. plausibility and faithfulness and compare them with attribution score extracted from fine-tuned models and large language models. |
| Outcome: | The proposed model outperforms attention and Integrated Gradients in plausibility and faithfulness, while fine-tuning models are harder to explain in low-resource settings. |
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions (2020.acl-main)
Copied to clipboard
| Challenge: | Modern deep learning models for NLP are notoriously opaque, and this has motivated efforts to design example-specific approaches to interpret such models. |
| Approach: | They propose to use influence functions to explain models by highlighting important words in input text to provide models with an explanation. |
| Outcome: | The proposed approach is particularly useful for natural language inference, a task in which ‘saliency maps’ may not have clear interpretation. |
JI2S: Joint Influence‐Aware Instruction Data Selection for Efficient Fine‐Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior selection strategies score samples using generalpurpose LLMs, leveraging their strong language understanding but introducing inherent biases that misalign with the target model’s behavior and yield unstable downstream performance. |
| Approach: | They propose a framework that jointly models marginal and combinatorial influences within sample groups and evaluate them on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
| Outcome: | The proposed framework outperforms fulldataset training and strong baselines on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit surprising abilities across a variety of language tasks. |
| Approach: | They propose an algorithm which selects a coreset by analyzing correlation between training and evaluation samples with a trained model. |
| Outcome: | The proposed algorithm can achieve similar performance with just 50% of the training data while preserving the accuracy of the existing model. |
Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Model (LLM) based multi-agent systems (MAS) have high potential for tackling complex tasks through collaborative intelligence. |
| Approach: | They propose a framework that incorporates influence scores to guide tree search and data selection in data synthesis. |
| Outcome: | The proposed framework incorporates influence scores to guide tree search and data selection in data synthesis. |