ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges (2025.findings-emnlp)
Copied to clipboard
Cheng Qian, Hongyi Du, Hongru Wang, Xiusi Chen, Yuji Zhang, Avirup Sil, ChengXiang Zhai, Kathleen McKeown, Heng Ji
| Challenge: | Existing benchmarks for large language models fail to reflect real-world complexity . existing benchmarks often fail to capture real-life problems . |
| Approach: | They propose a benchmark that features real-world-inspired, open-ended problems from competitions . they propose 'ModelingBench' that supports multiple valid solutions . |
| Outcome: | The proposed framework outperforms baselines and produces well-grounded, creative solutions. |
Similar Papers
LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, but their proficiency in mathematical reasoning remains a key challenge. |
| Approach: | They propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
| Outcome: | The proposed framework evaluates LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub (2025.acl-long)
Copied to clipboard
Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Cheng Qian, Zihe Wang, Yujia Qin, Yining Ye, Yaxi Lu, Chen Qian, Zhong Zhang, Yukun Yan, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing approaches lack flexibility to address diverse and ever-evolving user queries in open domains. |
| Approach: | They propose to evaluate LLMs on open-domain knowledge that requires tools to solve diverse and ever-evolving user queries. |
| Outcome: | The proposed system outperforms baselines in the open domain task-solving benchmark. |
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024.findings-acl)
Copied to clipboard
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Recent advances in large language models have showcased significant improvements in mathematics, but traditional benchmarks like GSM8k offer a unidimensional perspective. |
| Approach: | MathBench is a benchmark that rigorously assesses the mathematical capabilities of large language models. |
| Outcome: | MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, Jiaxuan You
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. |
| Approach: | They propose a benchmark to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. |
| Outcome: | The proposed framework measures task completion and quality of collaboration and competition using novel, milestone-based key performance indicators. |
ACEBench: A Comprehensive Evaluation of LLM Tool Usage (2025.findings-emnlp)
Copied to clipboard
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, Wu Liu
| Challenge: | Existing benchmarks for evaluating LLMs’ tool usage face several limitations: limited evaluation scenarios, lacking assessments in real multi-turn dialogue contexts; narrow evaluation dimensions, with insufficient detailed assessments of how LLM use tools; and reliance on LLM or real API executions for evaluation, which introduces significant overhead. |
| Approach: | ACEBench is a benchmark for evaluating tool usage in Large Language Models . it categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
| Outcome: | ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing agentic training data are narrow in task variety and easily solved . real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. |
| Approach: | They propose a framework that synthesizes diverse tool-use training data and simulates complete environments. |
| Outcome: | The proposed framework synthesizes diverse tool-use training data and simulates complete environments. |
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning. |
| Approach: | They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions. |
| Outcome: | The proposed frameworks are used to support reliable off-the-shelf AP planners. |
A Parallelized Framework for Simulating Large-Scale LLM Agents with Realistic Environments and Interactions (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing work on large language models lacks a realistic environment and parallelized framework to support complex interactions between agents and environments. |
| Approach: | They propose a framework that integrates realistic societal environments and parallelized interactions to support simulations of large-scale agents. |
| Outcome: | The proposed framework can support simulations of 30,000 agents faster than the wall-clock time with 24 NVIDIA A800 GPUs and the performance increases linearly with the increase of LLM computational resources. |