Papers with IT
Call for Rigor in Reporting Quality of Instruction Tuning Data (2025.acl-short)
Copied to clipboard
| Challenge: | Instruction tuning is crucial for adapting large language models (LLMs) to user intentions. |
| Approach: | They propose to use hyperparameters for training models that are often selected arbitrarily without adequate justification to make arbitrary conclusions. |
| Outcome: | The results show that arbitrary hyperparameter decisions can make any arbitrary conclusion. |
JI2S: Joint Influence‐Aware Instruction Data Selection for Efficient Fine‐Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior selection strategies score samples using generalpurpose LLMs, leveraging their strong language understanding but introducing inherent biases that misalign with the target model’s behavior and yield unstable downstream performance. |
| Approach: | They propose a framework that jointly models marginal and combinatorial influences within sample groups and evaluate them on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
| Outcome: | The proposed framework outperforms fulldataset training and strong baselines on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024.emnlp-main)
Copied to clipboard
Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, JingBo Zhu
| Challenge: | Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset. |
| Approach: | They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR). |
| Outcome: | The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations. |
A Multilingual Wikified Data Set of Educational Material (L18-1)
Copied to clipboard
Iris Hendrickx, Eirini Takoulidou, Thanasis Naskos, Katia Lida Kermanidis, Vilelmini Sosoni, Hugo de Vos, Maria Stasimioti, Menno van Zaanen, Panayota Georgakopoulou, Valia Kordoni, Maja Popovic, Markus Egg, Antal van den Bosch
| Challenge: | a crowdsourcing effort to annotate and link parallel texts has been unsuccessful . a data set of parallel texts in eleven languages is presented . |
| Approach: | They present a wikified data set of English sentences linked to Wikipedia pages . they use crowdsourcing to annotate the texts and perform crowdsourcing for complex annotations . |
| Outcome: | The proposed data set is valuable as it constitutes a rich resource . it includes annotated data of English sentences linked to translations in eleven languages . |
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning is effective in instruction learning for unseen tasks, but it relies on a large amount of human-annotated samples, which restricts its generalization. |
| Approach: | They propose an instruction tuning technique which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions and then tests its generalization ability on unseen tasks. |
| Outcome: | The proposed method improves IT performance versus labeled data and training tasks by constructing pseudo-labeled data from unlabele . data is used to build a model that can learn from human instructions for zero-shot generalization on unseen tasks. |
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models on diverse tasks with instructions. |
| Approach: | They propose a framework to identify informative tasks and then actively tune models on selected tasks. |
| Outcome: | The proposed method outperforms baseline strategies for task selection on NIV2 and Self-Instruct datasets. |
Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies on instruction tuning (IT) have achieved great performance with zero-shot generalizability to unseen tasks. |
| Approach: | They analyze how models utilize instructions during IT by comparing model training with altered vs. original instructions. |
| Outcome: | The proposed model outperforms naive models in low resource setting. |
Translation of Multifaceted Data without Re-Training of Machine Translation Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a novel MT pipeline that considers the intra-data relation is proposed . previous MT systems have demonstrated relatively low performance, making them hardly utilized as another data source. |
| Approach: | They propose a new MT pipeline that considers the intra-data relation . they propose CS and IT to enhance the intra data relation based on a data point . |
| Outcome: | The proposed pipeline improves translation quality and training data compared with the existing approach . it yields better training data and better translation quality than previous approaches . |
Exploring the Relationship between In-Context Learning and Instruction Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications, but they are significantly different. |
| Approach: | They examine how the hidden states of Large Language Models change in these two paradigms by examining how they differ in implementation. |
| Outcome: | The proposed model changes the hidden states of LLMs as if its accompanying demonstrations were used to instructionally tune the model. |
Retrieval Augmented Instruction Tuning for Open NER with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies have focused on integrating large language models (LLMs) with information extraction (IE) however, the best approach to incorporate information with LLMs for IE remains an open question. |
| Approach: | They propose to use a Chinese IT dataset to perform RA-IT for IE . they use semantically similar examples from the training dataset as the context . |
| Outcome: | The proposed approach is evaluated in English and Chinese scenarios. |
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction (2025.findings-acl)
Copied to clipboard
Yuxin Jiang, Yufei Wang, Chuhan Wu, Xinyi Dai, Yan Xu, Weinan Gan, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
| Challenge: | Existing methods for generating and curating high-quality instruction-tuning data rely heavily on the quality of seed data or strong assumptions about the structure and content of web documents. |
| Approach: | They propose a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. |
| Outcome: | The proposed framework outperforms state-of-the-art baselines by 16.65% across four instruction-following benchmarks. |
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (2024.emnlp-main)
Copied to clipboard
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha
| Challenge: | We propose a novel large-scale audio-language model with advanced audio understanding and reasoning abilities. |
| Approach: | They propose a general-purpose large audio-language model with advanced audio understanding and reasoning abilities that integrates an LLM with multiple types of audio representations. |
| Outcome: | The proposed model outperforms existing models on audio understanding tasks by 1%-84%. |
CITB: A Benchmark for Continual Instruction Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for instruction tuning do not leverage the rich natural language instructions. |
| Approach: | They propose to use a benchmark to study how instruction tuning works in CL tasks. |
| Outcome: | The proposed method can achieve similar or better results than existing CL methods. |
LimaCost: Data Valuation for Instruction Tuning of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction tuning is an effective approach for aligning large language models with human intentions. |
| Approach: | They propose a data quality measure that exhibits a strong correlation with model performance. |
| Outcome: | The proposed measure exhibits a strong correlation with model performance. |
Curriculum Consistency Learning for Conditional Sentence Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Consistency learning (CL) has proven to be a valuable technique for improving the robustness of conditional sentence generation models. |
| Approach: | They propose a strategy that guides models to learn consistency in alignment with their current capacity to differentiate between features. |
| Outcome: | The proposed strategy delivers +2.0 accuracy point improvement compared with vanilla IT and +0.7 COMET scores over traditional CL methods in MT tasks. |