OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on Mathematical Programming and Combinatorial Optimization, hindering comprehensive evaluation. |
| Approach: | They propose a benchmarking tool that compares 1,000 curated optimization problems across three difficulty levels. |
| Outcome: | The proposed model improves performance on hard problems while maintaining 27% accuracy. |
Similar Papers
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | NLCO evaluates large language models for combinatorial optimization (CO) . existing evaluations emphasize relatively simple reasoning competencies . |
| Approach: | They propose a combinatorial optimization benchmark that evaluates large language models on CO reasoning. |
| Outcome: | The proposed model can handle combinatorial optimization without writing code or calling external solvers. |
Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past decade. |
| Approach: | They propose a benchmark dataset for evaluating the problem solving abilities of large language models (LLMs) they curate 515 challenging problems from the highly competitive IIT JEE-Advanced exam. |
| Outcome: | The proposed model performs better on open-source and proprietary models than the current model, but with techniques like self-consistency, self-refinement and chain-of-thought prompting. |
OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and tool use, but their ability to continuously refine solutions in response to dynamic environmental feedback remains underexplored. |
| Approach: | They propose a benchmark to evaluate self-improvement capabilities in large-scale search spaces by combining 20 machine learning tasks with 10 classic NP-hard problems. |
| Outcome: | The proposed framework emulates human-like cognitive adaptation and operates via a general perception–memory–reasoning loop, iteratively refining solutions based on environmental feedback. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Comp-Comp is an iterative benchmarking framework grounded in the principles of comprehensiveness and compactness. |
| Approach: | They propose a benchmark framework that incorporates the principle of comprehensiveness and compactness. |
| Outcome: | The proposed framework is domain-agnostic and adaptable to a wide range of specialized fields. |
EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving (2026.findings-acl)
Copied to clipboard
Xiyuan Zhou, Xinlei Wang, Yirui He, Ruixi Zou, Yang Wu, Yuheng Cheng, Yulu Xie, Wenxuan Liu, Huan Zhao, Yan Xu, Jinjin Gu, Junhua Zhao
| Challenge: | Existing benchmarks focus on well-defined or abstract reasoning and fail to capture real-world engineering problems. |
| Approach: | They propose a hierarchical benchmark to evaluate large language models on engineering problems. |
| Outcome: | The proposed model performs well under well-defined conditions and is based on three levels of difficulty and covers diverse engineering subfields. |
LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences. |
| Approach: | They propose a framework which extends established benchmarks to sequential problem-solving settings and provides feedback after each round to build a demonstration memory that the models can query in future tasks. |
| Outcome: | The proposed framework can improve performance of LLMs by learning from past interactions and improve models' performance over time. |
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect the evolution of large language models (LLMs). |
| Approach: | They propose to use a Segment Policy Optimization algorithm to enhance the LLM's ability to accurately fulfill multi-task workflows. |
| Outcome: | The proposed benchmarks show that existing benchmarks lack the complexity required to fully reflect the evolution of large language models. |
XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) lack advanced capabilities such as temporal reasoning, future forecasting, and numerical modeling. |
| Approach: | They propose a benchmarking tool to evaluate LLMs' ability to solve complex financial problems across diverse graduate-level finance topics with multi-modal context. |
| Outcome: | The proposed model improves on the o1 model but still lags behind human experts in temporal reasoning and scenario planning capabilities. |