EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect the evolution of large language models (LLMs). |
| Approach: | They propose to use a Segment Policy Optimization algorithm to enhance the LLM's ability to accurately fulfill multi-task workflows. |
| Outcome: | The proposed benchmarks show that existing benchmarks lack the complexity required to fully reflect the evolution of large language models. |
Similar Papers
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs (2025.acl-long)
Copied to clipboard
Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. |
| Approach: | They propose a Chinese Comprehensive Constraints Following Benchmark for LLMs that compiles constraints from real-world instructions and constructs a systematic framework for constraint types. |
| Outcome: | The proposed framework integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. |
ULTRABENCH: Benchmarking LLMs under Extreme Fine-grained Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks evaluate models on only a few attributes, typically fewer than five . a new benchmark evaluates large language models under dense, multi-attribute constraints . |
| Approach: | They propose a benchmark that evaluates large language models under dense, multi-attribute constraints. |
| Outcome: | The proposed benchmark evaluates large language models under dense, multi-attribute constraints. |
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models (2024.acl-long)
Copied to clipboard
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, Wei Wang
| Challenge: | Existing benchmarks focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. |
| Approach: | They propose a Multi-level Fine-grained Constraints Following Benchmark for Large Language Models that adds a single constraint to the initial instruction at each increased level. |
| Outcome: | The proposed model can follow instructions with more constraints, and is deemed to have better instruction-following ability. |
CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation (2025.acl-industry)
Copied to clipboard
| Challenge: | CodeIF assesses the ability of large language models to adhere to task-oriented instructions in code generation tasks. |
| Approach: | They introduce a benchmark designed to assess LLMs' ability to adhere to task-oriented instructions within diverse code generation scenarios. |
| Outcome: | The proposed benchmark assesses LLMs' ability to adhere to task-oriented instructions in code generation tasks across a wide range of complexity levels and programming domains. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
C²RBench: A Chinese Complex Reasoning Benchmark for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks often fail to capture complex multi-step reasoning demands inherent in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate multi-step, multimodal advanced reasoning of large language models. |
| Outcome: | The proposed benchmark exceeds existing benchmarks in cognitive complexity and accuracy by over 90% . it features 1,115 carefully curated Chinese tasks organized into eight domain-specific subsets . evaluations of 20 LLMs and 24 multimodal large language models reveal critical performance gaps . |
XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) lack advanced capabilities such as temporal reasoning, future forecasting, and numerical modeling. |
| Approach: | They propose a benchmarking tool to evaluate LLMs' ability to solve complex financial problems across diverse graduate-level finance topics with multi-modal context. |
| Outcome: | The proposed model improves on the o1 model but still lags behind human experts in temporal reasoning and scenario planning capabilities. |
WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have developed watermarking algorithms which restrict the generation process to leave an invisible trace for watermark detection. |
| Approach: | They propose a benchmarking procedure that compares different methods to ensure consistent watermarking strength and jointly evaluates their generation and detection performance. |
| Outcome: | The proposed benchmark compares 4 open-source watermarks on 2 LLMs under 2 watermarking strengths and observes the common struggles for current methods on maintaining the generation quality. |
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models (2024.findings-acl)
Copied to clipboard
Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Noah Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, Wangchunshu Zhou, Yiming Liang, Lei Zhang, Lei Ma, Jiajun Zhang, Zuowen Li, Wenhao Huang, Chenghua Lin, Jie Fu
| Challenge: | a recent study shows that large language models have limited generalization in low-resource languages like Chinese. |
| Approach: | They propose to evaluate the zero-shot generalizability of large language models to the Chinese language . they release only half of the dataset publicly, with the remainder kept private . |
| Outcome: | The Chinese Instruction-Following Benchmark evaluates the generalizability of LLMs to the Chinese language. |
InFoBench: Evaluating Instruction Following Ability in Large Language Models (2024.findings-acl)
Copied to clipboard
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, Dong Yu
| Challenge: | Existing methods for evaluating Large Language Models (LLMs) ability to follow instructions have not been able to provide a detailed analysis of their compliance with instructions. |
| Approach: | They propose a new metric for evaluating Large Language Models' ability to follow instructions and a benchmark for DRFR. |
| Outcome: | The proposed metric and benchmark compared with traditional scoring methods and explores annotation sources including human experts, crowd-sourced workers, and GPT-4. |