| Challenge: | a lack of transparency has resulted in a gap between research community and leading companies . large language models have demonstrated remarkable capabilities in following complex instructions . |
| Approach: | They propose a method to build large language models that can follow complex instructions with open-source data. |
| Outcome: | The proposed approach can synergize complex instructions and filter responses with evaluation questions. |
Similar Papers
From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) follow instructions with elaborate requirements, yet it remains under-explored how to enhance their ability to follow complex instructions with multiple constraints. |
| Approach: | They propose a method to obtain and utilize effective training data to enhance LLMs' ability to follow complex instructions with multiple constraints. |
| Outcome: | The proposed framework improves models' ability to follow instructions generally and generalize effectively across out-of-domain, in domain, and adversarial settings while maintaining general capabilities. |
AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following (2026.acl-long)
Copied to clipboard
Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Kumar Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Hassan Awadalla, Manaal Faruqui
| Challenge: | Recent advances in large language models (LLMs) have shown impressive performance on a range of tasks, yet advanced instruction following (IF) remains a significant challenge. |
| Approach: | They propose a benchmark that features over 1,600 prompts and expert-curated rubrics that assess LLMs’ ability to follow complex, multi-turn, and system-level instructions. |
| Outcome: | The proposed framework improves instruction-following abilities of large language models, achieving a 6.7% gain on AdvancedIF and strong results on public benchmarks. |
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing algorithms to improve the ability of LLMs to follow complex instructions are lacking. |
| Approach: | They propose a benchmark to improve the ability to follow complex instructions by using a IOPO alignment method to take input and output preference into consideration. |
| Outcome: | The proposed algorithm shows 8.15%, 2.18% improvements on in-domain data and 5.91%, 2.83% on out-of-domain datasets compared to SFT and DPO respectively. |
CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation (2025.acl-industry)
Copied to clipboard
| Challenge: | CodeIF assesses the ability of large language models to adhere to task-oriented instructions in code generation tasks. |
| Approach: | They introduce a benchmark designed to assess LLMs' ability to adhere to task-oriented instructions within diverse code generation scenarios. |
| Outcome: | The proposed benchmark assesses LLMs' ability to adhere to task-oriented instructions in code generation tasks across a wide range of complexity levels and programming domains. |
FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies explore approaches to synthesize instruction data with open-sourced LLMs but require high-quality human-crafted seed data. |
| Approach: | They propose an end-to-end framework to synthesize high-quality instruction data with open-sourced LLMs and sampled unlabeled documents, eliminating the need for seed data. |
| Outcome: | The proposed framework synthesizes high-quality instruction data with open-sourced LLMs and sampled unlabeled documents, eliminating the need for seed data. |
EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect the evolution of large language models (LLMs). |
| Approach: | They propose to use a Segment Policy Optimization algorithm to enhance the LLM's ability to accurately fulfill multi-task workflows. |
| Outcome: | The proposed benchmarks show that existing benchmarks lack the complexity required to fully reflect the evolution of large language models. |
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles (2026.acl-long)
Copied to clipboard
Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu
| Challenge: | Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints. |
| Approach: | They propose a framework that uses tiny language models to evaluate instruction following . they propose to use a set of specialized tiny language model to provide rewards for soft constraints. |
| Outcome: | The proposed framework outperforms baseline models by 12% and speeds up training time by 3. |
ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior studies assess instruction adherence in the model’s main responses, but it is also critical for large reasoning models to follow user instructions throughout their reasoning process. |
| Approach: | They propose a systematic benchmark for assessing reasoning instruction following to assess the model's adherence to instructions. |
| Outcome: | The proposed benchmark reduces the risk of undesirable shortcuts, hallucinations, or reward hacking within reasoning traces. |
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs (2025.acl-long)
Copied to clipboard
Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. |
| Approach: | They propose a Chinese Comprehensive Constraints Following Benchmark for LLMs that compiles constraints from real-world instructions and constructs a systematic framework for constraint types. |
| Outcome: | The proposed framework integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. |
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models (2024.acl-long)
Copied to clipboard
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, Wei Wang
| Challenge: | Existing benchmarks focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. |
| Approach: | They propose a Multi-level Fine-grained Constraints Following Benchmark for Large Language Models that adds a single constraint to the initial instruction at each increased level. |
| Outcome: | The proposed model can follow instructions with more constraints, and is deemed to have better instruction-following ability. |