Sparse Activation Editing for Reliable Instruction Following in Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks fail to capture the challenges of instruction following in complex narrative contexts. |
| Approach: | They propose a training-free framework that identifies and edits instruction-relevant neurons using only natural language instructions without requiring labelled data. |
| Outcome: | The proposed framework improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data. |
Similar Papers
From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) follow instructions with elaborate requirements, yet it remains under-explored how to enhance their ability to follow complex instructions with multiple constraints. |
| Approach: | They propose a method to obtain and utilize effective training data to enhance LLMs' ability to follow complex instructions with multiple constraints. |
| Outcome: | The proposed framework improves models' ability to follow instructions generally and generalize effectively across out-of-domain, in domain, and adversarial settings while maintaining general capabilities. |
DecIF: Improving Instruction-Following through Decomposition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to obtain high-quality instruction-following data rely heavily on existing documents and existing methods. |
| Approach: | They propose a data synthesis framework, DecIF, which automatically generates accurate and diverse instruction-following data from scratch for supervised fine-tuning and reinforcement learning. |
| Outcome: | Extensive experiments show that the proposed framework can synthesize accurate instruction-following data for both SFT and RL paradigms compared to baselines. |
Advancing Language Models through Instruction Tuning: Recent Progress and Challenges (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | tutorial addresses three critical questions within the field of instruction tuning: (1) What are the current focal points in instruction tuning research? (2) What are best practices in training an instruction-following model? (3) What new challenges have emerged? |
| Approach: | This tutorial presents a systematic overview of recent advances in instruction tuning. |
| Outcome: | The tutorial covers different stages in model training: supervised fine-tuning, preference optimization, and reinforcement learning. |
Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models (2024.emnlp-main)
Copied to clipboard
Zhengxuan Wu, Yuhao Zhang, Peng Qi, Yumo Xu, Rujun Han, Yian Zhang, Jifan Chen, Bonan Min, Zhiheng Huang
| Challenge: | Modern language models fail to follow human instructions while being faithful . a trade-off exists between instruction following and faithfulness when training LMs . |
| Approach: | They propose a method that relies on Reject-sampling by Self-instruct with Continued Fine-tuning to train LMs to follow human instructions while being faithful. |
| Outcome: | The proposed method outperforms vanilla MTL with high-quality data, but with significantly smaller data. |
Learning to Follow Object-Centric Image Editing Instructions Faithfully (2023.findings-emnlp)
Copied to clipboard
| Challenge: | avrahami et al., 2022b,a): natural language instructions are often underspecified, requiring models to uncover their implicit meaning. |
| Approach: | They propose to use paired data to model the implicit meaning of instructions . they also propose to ground the model to localize where the edit has to be performed . |
| Outcome: | The proposed model performs better than state-of-the-art baselines on paired data, showing improvements in quality and faithfulness. |
Instruction Position Matters in Sequence Generation with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. |
| Approach: | They propose to shift the position of task instructions after the input sentences to enhance the model's instruction-following capability. |
| Outcome: | The proposed method outperforms traditional settings across various model scales (1B / 7B & 13B) and different sequence generation tasks (translation and summarization) without any additional data or annotation costs. |
Robust and Scalable Model Editing for Large Language Models (2024.lrec-main)
Copied to clipboard
Yingfa Chen, Zhengyan Zhang, Xu Han, Chaojun Xiao, Zhiyuan Liu, Chen Chen, Kuai Li, Tao Yang, Maosong Sun
| Challenge: | Existing methods that ignore contextual knowledge fail to reliably fall back to parametric knowledge when presented with irrelevant context. |
| Approach: | They propose to use contextual knowledge to update and correct LLMs' knowledge by in-context editing instead of retraining. |
| Outcome: | The proposed method outperforms current state-of-the-art methods by a large margin on a dataset that contains irrelevant questions. |
Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for ensuring Large Language Models (LLMs) follow complex instructions fail to reflect real-world use or isolate compliance from task success. |
| Approach: | They propose a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of LLM instruction compliance. |
| Outcome: | The proposed framework reveals that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. |
Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering (2026.eacl-long)
Copied to clipboard
| Challenge: | In many real-world applications, users rely on natural language instructions to guide large language models (LLMs) However, LLMs do not attend to these instructions reliably, and users lack simple mechanisms to emphasize their importance beyond modifying prompt wording or structure. |
| Approach: | They propose an inference-time method that enables users to emphasize specific parts of their prompt by steering the model’s attention toward them, aligning the model's perceived importance of different tokens with user intent. |
| Outcome: | The proposed method improves instruction following across tasks involving multiple instructions and generalizes across models of varying scales. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |