Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, Dongkuan Xu
| Challenge: | Existing frameworks for Augmented Language Models lack flexibility, democratization, and holistic evaluation. |
| Approach: | They propose a lightweight and extensible framework for Augmented Language Models called Gentopia. |
| Outcome: | The proposed framework integrates language models, task formats, prompting modules, and plugins into a unified paradigm. |
Similar Papers
Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are generalist agents capable of operating within complex environments. |
| Approach: | They propose a class of tools that can serve as a middleware layer shielding LLMs from environmental complexity. |
| Outcome: | The proposed tool can shield the LLM from environmental complexity in two representative complex environments. |
ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models (2023.emnlp-demo)
Copied to clipboard
Chenliang Li, He Chen, Ming Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu, Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, Hongzhu Shi, Ji Zhang, Fei Huang, Jingren Zhou
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior. |
| Approach: | They propose a framework that equips large language models with tool-use capabilities . they propose LLaMA and Chat-GLM as controllers, and a model-based agent framework . |
| Outcome: | The proposed framework equips open-source LLMs with tool-use capabilities . it provides a user-friendly system library with a customizable engine design . |
ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing frameworks for large language model embeddings have limited support for only a limited range of architectures and fine-tuning strategies. |
| Approach: | They propose a framework that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
| Outcome: | The proposed framework enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann
| Challenge: | Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios. |
| Approach: | They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks. |
| Outcome: | The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Augmented Language Models (LLMs) over-rely on task-specific demonstrations that limits their generalizability and computational cost. |
| Approach: | They propose a query-tool grounding algorithm that is generalizable to various tasks . they delegate tool grounding and execution to small language models and LLMs . |
| Outcome: | The proposed algorithm outperforms baselines on 14 datasets and shows it can be generalized to different tasks. |
Empowering Large Language Models for Textual Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | True. True. False |
| Approach: | False slants are proposed to generate a large pool of augmentation instructions and select the most suitable task-informed instructions. |
| Outcome: | False omissions: the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges (2025.findings-emnlp)
Copied to clipboard
Cheng Qian, Hongyi Du, Hongru Wang, Xiusi Chen, Yuji Zhang, Avirup Sil, ChengXiang Zhai, Kathleen McKeown, Heng Ji
| Challenge: | Existing benchmarks for large language models fail to reflect real-world complexity . existing benchmarks often fail to capture real-life problems . |
| Approach: | They propose a benchmark that features real-world-inspired, open-ended problems from competitions . they propose 'ModelingBench' that supports multiple valid solutions . |
| Outcome: | The proposed framework outperforms baselines and produces well-grounded, creative solutions. |
ATLAS: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning (2026.findings-acl)
Copied to clipboard
Jinyang Wu, Guocheng Zhai, Ruihan Jin, Jiahao Yuan, Yuhao Shen, Shuai Zhang, Zhengqi Wen, Jianhua Tao
| Challenge: | Existing approaches to optimize large language models with external tools are limited. |
| Approach: | They propose a dual-path framework for dynamic tool usage in cross-domain complex reasoning . they exploit empirical priors for domain-specific alignment and RL-based multi-step routing . |
| Outcome: | The proposed framework outperforms closed-source models and existing methods on in-distribution and out-of-distortion tasks. |