Can Language Models Follow Multiple Turns of Entangled Instructions? (2025.findings-emnlp)
Copied to clipboard
Chi Han, Xin Liu, Haodong Wang, Shiyang Li, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Qingyu Yin, Liang Qiu, Changlong Yu, Yifan Gao, Zheng Li, Bing Yin, Jingbo Shang, Heng Ji
| Challenge: | Despite of significant achievements in improving instruction-following capabilities of large language models, the ability to process multiple potentially entangled or conflicting instructions remains a considerable challenge. |
| Approach: | They construct multi-turn instruction with 1.1K high-quality multi-turned conversations using the human-in-the-loop approach and examine their capabilities. |
| Outcome: | The proposed model shows that it is difficult to integrate multiple turns and balance competing objectives when instructions intersect or conflict. |
Similar Papers
One Battle After Another: Probing LLMs’ Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for instruction-following in multi-topic dialogues are limited to a fixed number of turns, susceptible to saturation and failing to account for users’ interactive experience. |
| Approach: | They propose a framework featuring a three-layer tracking mechanism and a query synthesis agent to mimic sequential user behaviors. |
| Outcome: | The proposed framework outperforms existing benchmarks in the evaluation of instruction following in multi-topic dialogues and demonstrates deficiencies in failure recovery and fine-grained instruction following. |
Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models (2024.acl-long)
Copied to clipboard
Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Xin Zhao, Fuzheng Zhang, Di Zhang, Kun Gai
| Challenge: | Existing studies overlook the multi-turn instruction following ability of large language models (LLMs) Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi- turn instruction following. |
| Approach: | They propose a method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis, and a context-aware preference optimization strategy to further enhance LLMs for complex queries. |
| Outcome: | The proposed method improves existing LLMs by up to 7.2% in multi-turn instruction following. |
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions (2025.acl-long)
Copied to clipboard
Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici
| Challenge: | Large Language Models excel at solving individual problems in isolation, but are they able to effectively collaborate over long-term interactions? |
| Approach: | They propose to use a multi-session dataset to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. |
| Outcome: | The proposed model performs poorly when instructions are spread across sessions, suggesting that they are not able to integrate information over long interactions. |
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions. |
| Approach: | They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4. |
| Outcome: | The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information. |
From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) follow instructions with elaborate requirements, yet it remains under-explored how to enhance their ability to follow complex instructions with multiple constraints. |
| Approach: | They propose a method to obtain and utilize effective training data to enhance LLMs' ability to follow complex instructions with multiple constraints. |
| Outcome: | The proposed framework improves models' ability to follow instructions generally and generalize effectively across out-of-domain, in domain, and adversarial settings while maintaining general capabilities. |
When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following (2025.findings-emnlp)
Copied to clipboard
Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Edison Marrese-Taylor, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
| Challenge: | a large number of languages are increasingly used to evaluate their ability to follow multiple instructions simultaneously. |
| Approach: | They propose two benchmarks to evaluate LLMs' ability to follow multiple instructions simultaneously . they use many instruction-following eval and style-aware Mostly Basic programming problems . |
| Outcome: | The proposed models predict performance on unseen instruction combinations and not used during training with 10% error. |
Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are typically trained to follow a single instruction per inference call. |
| Approach: | They introduce a benchmark to evaluate Large language models' ability to follow one instruction per inference call. |
| Outcome: | The proposed model reduces the total inference time by 1.46 times in average since it does not require multiple inference calls. |
Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can perform a wide variety of language tasks when presented in English. |
| Approach: | They propose a method to evaluate the multilingual capabilities of large language models using a prompt back-translation method to find out how LLMs acquire their multilingual abilities. |
| Outcome: | The proposed method shows that large language models can transfer learned knowledge across different languages, but struggle to provide accurate results in translation-variant tasks. |
The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, Maarten Rijke
| Challenge: | Current evaluation resources for instruction following focus on single task instructions, but the instruction sequences in these benchmarks often lack coherence. |
| Approach: | They propose to evaluate models’ abilities to follow multiple instructions through sequential instruction following tasks using four tasks to assess different aspects of sequential instruction followed. |
| Outcome: | The proposed benchmark outperforms open-source and closed-source models on four tasks assessing different aspects of sequential instruction following. |
BotChat: Evaluating LLMs’ Capabilities of Having Multi-Turn Dialogues (2024.findings-naacl)
Copied to clipboard
Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Modern Large Language Models (LLMs) facilitate high-quality, multi-turn dialogues with humans, but human-based evaluation of such a capability requires substantial manual effort. |
| Approach: | They propose to evaluate LLMs' ability to emulate human-like, multi-turn conversations using an LLM-centric approach. |
| Outcome: | The proposed model emulates human-like, multi-turn conversations using an LLM-centric approach. |