APIS

Data2Text Studio: Automated Text Generation from Structured Data (D18-2)

Challenge:	a comprehensive understanding of the field is needed to maintain respectful and inclusive online environments.
Approach:	This tutorial aims to provide attendees with a comprehensive understanding of the field by delving into essential dimensions such as multilingualism, counter-narrative generation, a hands-on session with one of the most popular APIs for detecting hate speech, fairness, and ethics in AI, and the use of recent advanced approaches.
Outcome:	This tutorial aims to provide attendees with a comprehensive understanding of the field by delving into essential dimensions such as multilingualism, counter-narrative generation, a hands-on session with one of the most popular APIs for detecting hate speech, fairness, and ethics in AI, and the use of recent advanced approaches.

Copied to clipboard

Longxu Dou, Guanghui Qin, Jinpeng Wang, Jin-Ge Yao, Chin-Yew Lin

N-LTP: An Open-source Neural Language Technology Platform for Chinese (2021.emnlp-demo)

Challenge:	Data2Text Studio is a platform for automated text generation from structured data.
Approach:	They conduct experiments on RotoWire datasets for template extraction and text generation . they find that the Semi-HMMs model improves interactivity and interpretability .
Outcome:	The proposed model improves on template extraction and text generation tasks on RotoWire datasets.

Copied to clipboard

Wanxiang Che, Yunlong Feng, Libo Qin, Ting Liu

CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases (2025.naacl-long)

Challenge:	Existing tools that teach an independent model for each task are not supported in Chinese.
Approach:	They propose an open-source neural language platform supporting six Chinese NLP tasks . source code, documentation, and pre-trained models are available at https://github.com/hit-SCIR/ltp .
Outcome:	The proposed platform supports six Chinese NLP tasks.

Copied to clipboard

Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Qizhe Shieh, Wenmeng Zhou

TinyAgent: Function Calling at the Edge (2024.emnlp-demo)

Challenge:	Large Language Models excel in stand-alone code tasks but struggle with handling entire code repositories.
Approach:	They propose a system that integrates LLM agents with graph database interfaces extracted from code repositories.
Outcome:	The proposed system integrates LLM agents with graph database interfaces extracted from code repositories.

Copied to clipboard

Lutfi Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, Amir Gholami

Integrating INCEpTION into larger annotation processes (2024.emnlp-demo)

Challenge:	Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries.
Approach:	They propose an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge.
Outcome:	The proposed model outperforms existing models by reducing the input prompt length and quantizing the inference speed.

Copied to clipboard

Richard Eckart De Castilho, Jan-Christoph Klie, Iryna Gurevych

Challenge:	Annotation tools are increasingly only steps in a larger process into which they need to be integrated.
Approach:	They propose to adapt INCEpTION, a semantic annotation platform that offers intelligent assistance and knowledge management.
Outcome:	The proposed platform offers a range of APIs and can interact with external services such as authorization services, crowdsourcing platforms, terminology services or machine learning services.

Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems (2021.naacl-demos)

Copied to clipboard

Anish Acharya, Suranjit Adhikari, Sanchit Agarwal, Vincent Auvray, Nehal Belgamwar, Arijit Biswas, Shubhra Chandra, Tagyoung Chung, Maryam Fazel-Zarandi, Raefer Gabriel, Shuyang Gao, Rahul Goel, Dilek Hakkani-Tur, Jan Jezabek, Abhay Jha, Jiun-Yu Kao, Prakash Krishnan, Peter Ku, Anuj Goyal, Chien-Wei Lin, Qing Liu, Arindam Mandal, Angeliki Metallinou, Vishal Naik, Yi Pan, Shachi Paul, Vittorio Perera, Abhishek Sethi, Minmin Shen, Nikko Strom, Eddie Wang

SPRING Goes Online: End-to-End AMR Parsing and Generation (2021.emnlp-demo)

Challenge:	Traditional goal-oriented dialogue systems require annotations which are hard to obtain for every new domain, limiting scalability.
Approach:	They propose a data-driven approach to building goal-oriented dialogue systems . they use a seed dialogue simulator to generate annotated conversations instead of collecting annotations .
Outcome:	The proposed system improves turn-level action signature prediction accuracy by 50% . the system is scalable, extensible and data efficient .

Copied to clipboard

Rexhina Blloshmi, Michele Bevilacqua, Edoardo Fabiano, Valentina Caruso, Roberto Navigli

The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community (2025.acl-demo)

Challenge:	Abstract Meaning Representation (AMR) is a formalism for representing the semantics of natural language in a readable and hierarchical way.
Approach:	They present SPRING Online Services, a Web interface and RESTful APIs for their AMR parsing and generation system, SPRING (Symmetric PaRsIng aNd Generation).
Outcome:	The proposed system provides a highly interactive visualization platform and feedback mechanism to obtain user suggestions for further improvements of the system’s output.

Copied to clipboard

Shachar Don-Yehiya, Leshem Choshen, Omri Abend

Generating OpenAPI Specifications from Online API Documentation with Large Language Models (2025.acl-industry)

Challenge:	Using human-model conversations is a valuable resource for model development and research, but the open source and research community lags behind.
Approach:	They propose a unified set of human conversations with large language models and a plugin for voluntarily contributing user-model conversations.
Outcome:	The ShareLM collection and its plugin allow users to share conversations from most platforms.

Copied to clipboard

Koren Lazar, Matan Vetzler, Kiran Kate, Jason Tsay, David Boaz, Himanshu Gupta, Avraham Shinnar, Rohith D Vallam, David Amid, Esther Goldbraich, Jim Laredo, Ateret Anaby Tavor

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024.naacl-demo)

Challenge:	API specifications are often presented as unstructured HTML pages, requiring external users to manually convert it into a structured format.
Approach:	They propose a framework that transforms long API documentation pages into consistent, machine-readable API specifications.
Outcome:	The proposed framework generalizes well across hundreds of APIs and produces valid OpenAPI specifications that encapsulate most of the information from the original documentation.

Copied to clipboard

Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence

Challenge:	Existing benchmarks are narrow and simply compute overall task success.
Approach:	They propose a framework where both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs.
Outcome:	The proposed framework can track agent progress on two use cases and identify common failure points and refine the agent architecture to obtain a significant performance increase.

When Language Model Meets Private Library (2022.findings-emnlp)

Copied to clipboard

Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Wang Yongji, Jian-Guang Lou

Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation (2026.acl-demo)

Challenge:	Existing language models have been pre-trained on large-scale code corpora and generate decent code snippets.
Approach:	They propose a framework that can provide pre-trained language models with the ability to generate code using private libraries.
Outcome:	The proposed framework can generate code using private libraries using off-the-shelf language models or pre-trained models on code corpus containing API information.

Copied to clipboard

Abdelrahman Abdallah, Bhawna Piryani, Jamshid Mozafari, Andreas Herzinger, Jamie Holdcroft, Adam Jatowt

Information Retrieval

Challenge:	Rankify unifies retrieval-augmented generation (RAG) and retrieval based question answering systems.
Approach:	They propose an open-source Python toolkit that unifies retrieval-augmented generation in a single modular framework.
Outcome:	The proposed framework unifies retrieval-augmented generation (RAG) tools in a single modular framework.

LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models (2024.acl-demos)

Copied to clipboard

Abhishek Arora, Melissa Dell

Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning (2024.findings-naacl)

Challenge:	Large language models (LLMs) are used for many computational analyses, but approximate string matching packages are not widely used in social science applications.
Approach:	The open-source package LinkTransformer provides an end-to-end software for performing record linkage and other data cleaning tasks with transformer LLMs.
Outcome:	The open-source package LinkTransformer outperforms standard methods in a variety of languages and settings.

Copied to clipboard

Yinger Zhang, Hui Cai, Xierui Song, Yicheng Chen, Rui Sun, Jing Zheng

TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024.emnlp-industry)

Challenge:	Existing approaches to enable large language models to implement function calling are limited in their tool-use capabilities.
Approach:	They propose a controllable, target-driven approach to empower LLMs to operate external APIs only via prompts.
Outcome:	The proposed approach limits LLMs to executing simple tasks, e.g., API Selection and Argument Completion.

Copied to clipboard

Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang

FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs (2024.naacl-long)

Challenge:	Large language models have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools.
Approach:	They propose a framework to enhance the task planning and tool usage abilities of LLMs in industrial systems.
Outcome:	The proposed framework enhances the task planning and tool usage abilities of LLM-based agents in industrial systems.

Copied to clipboard

Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, Arshit Gupta

Challenge:	Flow-adhering planning algorithm for task oriented dialogs (TODs) is a task-oriented dialog (TO) that can be used for task planning and API usage.
Approach:	They propose a Flow-Adhering Planning algorithm that follows predefined flows and preserves API dependencies in task oriented dialogs.
Outcome:	The proposed algorithm outperforms other decoding and prompting-based baselines in task oriented dialogs.

Toolscaler: Scalable Generative Tool Calling via Structure-Aware Semantic Tokenization (2025.findings-emnlp)

Copied to clipboard

Yunyue Su, Zhang Jinshuai, Bowen Fang, Wen Ye, Jinghao Zhang, Bowen Song, Weiqiang Wang, Qiang Liu, Liang Wang

Syntactic Text Processing

Challenge:	Extensive experiments demonstrate the effectiveness of SGTC across various tasks.
Approach:	They propose a generative tool invocation framework that introduces structure-aware semantic tokenization to encode tools as discrete code sequences.
Outcome:	The proposed framework reduces the size of the representation space and underutilizes collaborative signals among tools in downstream tasks.

Project Debater APIs: Decomposing the AI Grand Challenge (2021.emnlp-demo)

Copied to clipboard

Roy Bar-Haim, Yoav Kantor, Elad Venezian, Yoav Katz, Noam Slonim

Challenge:	Project Debater is the first AI system that can debate human experts on complex topics.
Approach:	They describe Project Debater's architecture and evaluate its performance . they will focus on Key Point Analysis, a novel technology that identifies main points .
Outcome:	The proposed system can debate human experts on complex topics.

Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction (2023.acl-demo)

Copied to clipboard

Gabriele Picco, Marcos Martinez Galindo, Alberto Purpura, Leopold Fuchs, Vanessa Lopez, Thanh Lam Hoang

SummerTime: Text Summarization Toolkit for Non-experts (2021.emnlp-demo)

Challenge:	ZSL is a machine learning field that uses textual descriptions of entities or relations to perform tasks that are not seen during training.
Approach:	They propose a framework that allows researchers to compare state-of-the-art ZSL methods with standard benchmark datasets.
Outcome:	The proposed framework compares state-of-the-art methods with benchmark datasets and provides APIs for production under the standard SpaCy NLP pipeline.

Copied to clipboard

Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu, Ahmed Hassan Awadallah, Dragomir Radev

Text Generation

Challenge:	Recent advances in summarization provide models that can generate high quality summaries . a new toolkit for summarizing text is being developed to make it easier for non-experts to keep track of them.
Approach:	They develop a toolkit for text summarization that integrates with libraries designed for NLP researchers.
Outcome:	SummerTime is a toolkit for text summarization, including models, datasets, and evaluation metrics.

Prompt2Model: Generating Deployable Models from Natural Language Instructions (2023.emnlp-demo)

Copied to clipboard

Vijay Viswanathan, Chenyang Zhao, Amanda Bertsch, Tongshuang Wu, Graham Neubig

Low-Resource NLP

Challenge:	Large language models (LLMs) are a step backward from traditional special-purpose NLP models . they require extensive computational resources for deployment and can be gated behind APIs .
Approach:	They propose a general-purpose method that takes a natural language task description and uses it to train a special-purpose model.
Outcome:	The proposed method outperforms a strong LLM by 20% while being 700 times smaller.

entity-linkings: A Unified Library for Entity Linking (2026.eacl-demo)

Copied to clipboard

Yuya Sawada, Tsuyoshi Fujita, Yusuke Sakai, Taro Watanabe

Knowledge Representation

Challenge:	Entity linking (EL) is the task of mapping named entities in text to canonical entries in a knowledge base.
Approach:	They propose a unified library for using and developing entity linking systems . a strong emphasis is placed on usability, making it highly extensible .
Outcome:	a new library aims to disambiguate named entities in text by mapping them to canonical entries in a knowledge base.

Evaluating Embedding APIs for Information Retrieval (2023.acl-industry)

Copied to clipboard

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-hermelo, Mehdi Rezagholizadeh, Jimmy Lin

Petals: Collaborative Inference and Fine-tuning of Large Models (2023.acl-demo)

Challenge:	a growing number of language models are limiting their access to the community . we evaluate existing APIs for domain generalization and multilingual retrieval .
Approach:	They evaluate semantic embedding APIs in retrieval scenarios to assess their capabilities . they use BEIR and MIRACL to re-rank BM25 results using the APIs .
Outcome:	The proposed model is based on semantic embedding APIs that build vector representations of a given text.

Copied to clipboard

Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel

A Comparative Study on Schema-Guided Dialogue State Tracking (2021.naacl-main)

Challenge:	Recent studies show that pretrained language models can solve practical tasks using more than 100 billion parameters.
Approach:	They propose a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties.
Outcome:	The proposed system outperforms offloading for very large models running on consumer GPUs with 1 step per second, enough for many interactive LLM applications.

Copied to clipboard

Jie Cao, Yi Zhang

Natural Language Interfaces

Challenge:	Recent work proposes using natural language descriptions to define domain ontologies for dialog state tracking.
Approach:	They propose to use natural language descriptions to define domain ontologies instead of tag names for each intent or slot . they introduce a set of newly designed bench-marking descriptions and show model robustness .
Outcome:	The proposed model is robust on homogeneous and heterogeneously described descriptions in training and evaluation.

The olmOCR Project: Building Fully Open OCR using VLMs (2026.acl-demo)

Copied to clipboard

Jake Poznanski, Kyle Lo, Luca Soldaini

MapAgent: A Hierarchical Agent for Geospatial Reasoning with Dynamic Map Tool Integration (2026.findings-eacl)

Challenge:	olmOCR is a fully open OCR system developed through iterative public releases and community feedback.
Approach:	They propose an open OCR system that combines a 7B vision-language model trained in two stages: finetuning and reinforcement learning with visual unit tests.
Outcome:	The proposed system achieves state-of-the-art performance among open systems and proprietary APIs at a fraction of the cost.

Copied to clipboard

Md Hasebul Hasan, Mahir Labib Dihan, Tanzima Hashem, Mohammed Eunus Ali, Md Rizwan Parvez

Reasoning

Challenge:	Existing frameworks for large language models are tailored to domains such as mathematics, coding, or web automation.
Approach:	They propose a hierarchical multi-agent plug-and-play framework with customized toolsets and agentic scaffolds for map-integrated geospatial reasoning.
Outcome:	The proposed framework decouples planning from execution and reduces cognitive load on users.

Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models (2025.naacl-long)

Copied to clipboard

Danyang Liu, Fanjie Kong, Xiaohang Sun, Dhruva Patil, Avijit Vajpayee, Zhu Liu, Vimal Bhat, Najmeh Sadoughi

Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024.emnlp-industry)

Challenge:	Multimodal machine translation (MMT) aims to leverage additional modalities beyond text . current MMT systems rely heavily on monolingual English captioning data .
Approach:	They propose a reasoning-based framework to leverage large-scale vision-language models for MMT . they propose Detect, Disambiguate, and Translate framework to detect ambiguity in input sentence .
Outcome:	The proposed framework outperforms state-of-the-art models in disambiguation accuracy and translation quality.

Copied to clipboard

Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G P Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi

Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMs (2026.findings-acl)

Challenge:	Existing research explores the use of Large Language Models (LLMs) as the backbone of agentic systems.
Approach:	They propose a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling that has better generalizability on multiple tasks across seven evaluation benchmarks.
Outcome:	The proposed model outperforms more than 15 other models on out-of-domain datasets and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL).

Copied to clipboard

Zheyuan Liu, Liqiang Xiao, Yang Li, Hyokun Yun, Lihong Li, Chao Zhang, Meng Jiang

Extracted BERT Model Leaks More Information than You Think! (2022.emnlp-main)

Challenge:	Existing benchmarks primarily evaluate planning and execution success, overlooking the self-reflective dimension of tool use.
Approach:	They propose a benchmark to assess LLMs’ self-reflective reasoning in tool-augmented multi-turn dialogues.
Outcome:	The proposed benchmark covers 10 domains with 88 distinct APIs and 968 annotated dialogues, systematically injecting diverse error types arising from both user and assistant behavior.

Copied to clipboard

Xuanli He, Lingjuan Lyu, Chen Chen, Qiongkai Xu

Can’t Hide Behind the API: Stealing Black-Box Commercial Embedding Models (2025.findings-naacl)

Challenge:	Existing pre-trained language models are vulnerable to model extraction attacks . model extraction can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies.
Approach:	They propose to launch an attribute-inference attack against an extracted BERT model to prevent privacy leakage.
Outcome:	The proposed attack can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies.

Copied to clipboard

Manveer Singh Tamber, Jasper Xian, Jimmy Lin

GraphQL Query Generation: A Large Training and Benchmarking Dataset (2024.emnlp-industry)

Challenge:	a new study examines the retrieval effectiveness of commercial embedding models . robert mcgahey: can commercial embeds be "stolen" using distillation techniques? he says stealing models can offer benefits to different actors, including reduced costs and security .
Approach:	They propose to "steal" embedding models by training thief models on text–embedding pairs . they replicate retrieval effectiveness of commercial embeddable models with a cost of under $300 .
Outcome:	The proposed methods replicate retrieval effectiveness of commercial embedding models with under $300 . authors suggest measures to mitigate risk of model theft.

Copied to clipboard

Manish Kesarwani, Sambit Ghosh, Nitin Gupta, Shramona Chakraborty, Renuka Sindhgatta, Sameep Mehta, Carlos Eberhardt, Dan Debrunner

Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling (2026.eacl-long)

Challenge:	GraphQL is a powerful query language for APIs, but crafting complex GraphqL queries can be challenging.
Approach:	a team of researchers has created a large-scale, cross-domain text-to-GraphQL query operation dataset . the dataset includes 10,940 training triples spanning 185 cross-source data stores and 957 test triples over 14 data stores.
Outcome:	The proposed dataset includes 10,940 training triples and 957 test triples over 14 data stores.

Copied to clipboard

Benjamin Elder, Anupama Murthi, Jungkoo Kang, Ankita Rajaram Naik, Kiran Kate, Kinjal Basu, Danish Contractor

Model Extraction and Adversarial Transferability, Your BERT is Vulnerable! (2021.naacl-main)

Challenge:	Large language models rely on external tools and APIs to perform tasks specified in natural language.
Approach:	They propose a benchmark that transforms SQL queries from BIRD-SQL into executable API sequences.
Outcome:	The proposed benchmark evaluates 10 LLMs and 4 ReACT agents with low task completion rates and 50% task completion rate.

Copied to clipboard

Xuanli He, Lingjuan Lyu, Lichao Sun, Qiongkai Xu

PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction (2025.emnlp-industry)

Challenge:	Pretrained language models are used for natural language processing (NLP) but when they are deployed as a service, they can suffer from different attacks .
Approach:	They propose two defence strategies to protect the target model from adversarial attacks . they show that model extraction and adversarially transferable attacks can be effective .
Outcome:	The extracted model can lead to highly transferable adversarial attacks against the target model.

Copied to clipboard

Anubhav Shrimal, Aryan Jain, Soumyajit Chowdhury, Promod Yenigalla

Information Extraction & Text Mining

Challenge:	Structured information extraction from unstructured text is critical for Software 3.0 systems . current approaches to extract structured information from unstructed text are static contracts .
Approach:	They propose a system that automates JSON schemas for LLM consumption and optimizes them for LRM consumption.
Outcome:	The proposed system improves extraction accuracy and reduces errors by 92% within the first retry and maintaining practical latency.

CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels (2023.acl-long)

Copied to clipboard

Hyunsoo Cho, Youna Kim, Sang-goo Lee

Stepwise Reasoning Disruption Attack of LLMs (2025.acl-long)

Challenge:	Utilizing language models without internal access is becoming an attractive paradigm in the field of NLP . prompting has shown progressive performance enhancements in situations where data labels are scarce or unavailable.
Approach:	They propose a method that uses a weak-supervision signal to train a lightweight model without internal access to data labels.
Outcome:	The proposed method improves text classification accuracy with weak-supervision signal without accessing weights or gradients of the LM model or data labels.

Copied to clipboard

Jingyu Peng, Maolin Wang, Xiangyu Zhao, Kai Zhang, Wanyu Wang, Pengyue Jia, Qidong Liu, Ruocheng Guo, Qi Liu

APIRecX: Cross-Library API Recommendation via Pre-Trained Language Model (2021.emnlp-main)

Challenge:	Existing attacks on LLM reasoning are constrained by specific settings or lack of imperceptibility, limiting their feasibility and generalizability.
Approach:	They propose a stepwise rEasoning error disruption attack that subtly injects errors into prior reasoning steps to mislead the model into producing incorrect subsequent reasoning and final answers.
Outcome:	The proposed attack is compatible with zero-shot and few-shot settings, maintains the natural reasoning flow, and ensures covert execution without modification of the instruction.

Copied to clipboard

Yuning Kang, Zan Wang, Hongyu Zhang, Junjie Chen, Hanmo You

Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges (2025.findings-acl)

Challenge:	API recommendation tools can help programmers use APIs by recommending which APIs to be used next given the APIs that have been written.
Approach:	They propose a cross-library API recommendation approach that uses BPE to split API calls in each sequence and pre-train a GPT based language model.
Outcome:	The proposed APIRecX can recommend APIs that are previously regarded as OOV . it can migrate knowledge of existing libraries to a new library and recommend API that is previously viewed as OVO .

Copied to clipboard

Hongru Wang, Wenyu Huang, Yufei Wang, Yuanhao Xi, Jianqiao Lu, Huan Zhang, Nan Hu, Zeming Liu, Jeff Z. Pan, Kam-Fai Wong

Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)

Challenge:	Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use focus on stateless, single-turn interactions or partial evaluations, overlooking the inherent stateful nature of interactions in multi-turn applications.
Approach:	They propose a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use across six key tasks in three stages . they also build VirtualMobile – an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs.
Outcome:	The proposed dataset evaluates 13 open- and closed-source LLMs and provides detailed analysis at each stage.

Copied to clipboard

Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao

GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer (2024.naacl-long)

Challenge:	Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive .
Approach:	They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training .
Outcome:	The proposed framework achieves superior performance compared with baselines.

Copied to clipboard

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry Charnois

Named Entity Recognition

Challenge:	Named Entity Recognition (NER) models are limited to a set of predefined entity types. Large language models (LLMs) can extract arbitrary entities through natural language instructions.
Approach:	They propose a model that can identify any type of entity using a transformer encoder.
Outcome:	The proposed model outperforms existing models on NER benchmarks on a set of predefined entities.

MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents (2026.acl-long)

Copied to clipboard

Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, WenHao Wang, Yuehao Huang, Yuxiang Chai, Zhaolu Kang, Yaxuan Guo, Hao Wang, Kexin Zhang, Liang Liu, Yong Liu

Green & Sustainable NLP

Challenge:	Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, but systematic evaluation of GUI–shortcut hybrid agents remains underexplored.
Approach:	They propose a benchmark that evaluates GUI-shortcut hybrid agents with a specific focus on the mobile domain.
Outcome:	MAS-Bench evaluates agent's ability to generate shortcuts by discovering and creating reusable, low-cost workflows.

PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter (2023.emnlp-main)

Copied to clipboard

Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, Jing Xiao

AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents (2025.acl-long)

Challenge:	Large Language Models (LLMs) are too large to be fine-tuned with budget constraints and some are only accessible via APIs.
Approach:	They propose a pluggable Reward-Driven Contextual Adapter that integrates large language models as generators and trains them to refine the retrieved information.
Outcome:	The proposed method improves ReQA performance on three datasets by up to 20% compared to existing methods.

Copied to clipboard

Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

Green & Sustainable NLP

Challenge:	Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents’ performance in complex tasks.
Approach:	They propose a novel agent framework that prioritizes actions through application programming interfaces over UI actions and facilitates the creation and expansion of APIs through automated exploration of applications.
Outcome:	The proposed framework reduces task completion time by 65%-70% and cognitive workload by 38%-53% while maintaining accuracy of 97%-98% compared to humans.

CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions (2025.acl-long)

Copied to clipboard

Tamer Alkhouli, Katerina Margatina, James Gung, Raphael Shu, Claudia Zaghi, Monica Sunkara, Yi Zhang

Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models (2025.findings-acl)

Challenge:	Current benchmarks lack comprehensive assessment of large language models in complex conversational scenarios.
Approach:	They introduce a conversational benchmark to evaluate function-calling capabilities of large language models.
Outcome:	The proposed benchmark evaluates the function-calling capabilities and response quality of large language models.

Copied to clipboard

Xi Li, Ruofan Mao, Yusen Zhang, Renze Lou, Chen Wu, Jiaqi Wang

Continual Learning Using Only Large Language Model Prompting (2025.coling-main)

Challenge:	Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, but are vulnerable to backdoor attacks.
Approach:	They propose a chain-of-scrutiny approach which leverages LLMs’ unique reasoning abilities to mitigate backdoor attacks.
Outcome:	The proposed model is well-suited for the popular API-only LLM deployments, enabling detection at minimal cost and with little data.

Copied to clipboard

Jiabao Qiu, Zixuan Ke, Bing Liu

Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents (2025.findings-emnlp)

Challenge:	Existing continuous learning paradigms fine-tune language model parameters or use adapters or variants to adapt the LM.
Approach:	They propose a new continual learning paradigm wherein a large language model is regarded as a black box.
Outcome:	The proposed method outperforms baselines by a large margin in learning tasks incrementally.

Copied to clipboard

Pengzhou Cheng, Haowen Hu, Zheng Wu, Zongru Wu, Tianjie Ju, Daizong Ding, Zhuosheng Zhang, Gongshen Liu

KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server (2024.emnlp-main)

Challenge:	MLLM-powered GUI agents expose multiple interaction-level triggers, causing backdoor attacks . backdoor injection maximizes feature difference across sample classes, improving flexibility .
Approach:	They propose a framework for red-teaming backdoor attacks using MLLMs . they construct composite triggers by combining goal and interaction levels .
Outcome:	The proposed framework is effective and stealthy for red-teaming backdoor attacks.

Copied to clipboard

WenHao Wang, Xiaoyu Liang, Rui Ye, Jingyi Chai, Siheng Chen, Yanfeng Wang

META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI (2022.emnlp-main)

Challenge:	Existing methods to train large language models on private data are not effective because they rely on a local model for generation, resulting in a performance decline, or expose private data to API servers.
Approach:	They propose a client-server framework which enhances synthetic data quality and improves model performance while ensuring privacy.
Outcome:	The proposed framework improves model performance and privacy while learning local knowledge from the private data with differential privacy (DP) and distilling professional knowledge from server.

Copied to clipboard

Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, Kai Yu

CAVE: Controllable Authorship Verification Explanations (2025.naacl-long)

Challenge:	Current task-oriented dialogue systems focus on multi-turn text/speech interaction, then call back-end APIs to perform task.
Approach:	They propose a GUI-based task-oriented dialogue system that can perform GUI operations on real APPs without invoking TOD-specific backend APIs.
Outcome:	The proposed GUI-based task-oriented dialogue system can perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs.

Copied to clipboard

Sahana Ramnath, Kartik Pandey, Elizabeth Boschee, Xiang Ren

Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)

Challenge:	Authorship Verification (AV) is used for tasks such as plagiarism detection, forensic analysis, analysis of the spread of misinformation.
Approach:	They propose to train an offline authorship verification model that is accessible and easy to use.
Outcome:	The proposed model generates high quality explanations and competitive task accuracy on three difficult AV datasets.

Copied to clipboard

Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)

Challenge:	Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs.
Approach:	They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal.
Outcome:	The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks.

Copied to clipboard

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker

Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel (2024.emnlp-main)

Challenge:	Perception of toxicity evolves over time and differs between geographies and cultural backgrounds.
Approach:	They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results.
Outcome:	The Perspective API has been updated to reflect the changes in toxicity scores.

Copied to clipboard

Brendan King, Jeffrey Flanigan

Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents (2024.acl-long)

Challenge:	a task-oriented dialogue system requires turn-level annotations for interacting with their APIs.
Approach:	They propose an unsupervised approach that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent.
Outcome:	The proposed method doubles the success rate of a strong GPT-3.5 benchmark.

Copied to clipboard

Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang

I am a Strange Dataset: Metalinguistic Tests for Language Models (2024.acl-long)

Challenge:	Existing benchmarks for LLM-based mobile agents are insufficient to evaluate their capabilities.
Approach:	They propose a benchmark to evaluate LLM-based mobile agents' planning capabilities . they expand UI operations by incorporating 103 APIs to accelerate task completion .
Outcome:	The proposed benchmarks are based on 103 collected APIs and real user queries . the data is categorized into three distinct groups: SAST, SAMT, and MAMT .

Copied to clipboard

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

Template Guided Text Generation for Task-Oriented Dialogue (2020.emnlp-main)

Challenge:	Existing datasets for metalinguistic self-reference are limited by the number of subtasks.
Approach:	They propose a dataset that aims to address metalinguistic self-reference in large language models.
Outcome:	The proposed dataset is hand-crafted by experts and validated by non-expert annotators.

Copied to clipboard

Mihir Kale, Abhinav Rastogi

Self-Guided Function Calling in Large Language Models via Stepwise Experience Recall (2025.findings-emnlp)

Challenge:	a new study examines the use of templates to generate natural language utterances for a large number of APIs.
Approach:	They propose a schema-guided approach which conditions the generation on a natural language schema.
Outcome:	The proposed method improves over strong baselines, is robust to out-of-domain inputs and shows improved sample efficiency.

Copied to clipboard

Sijia Cui, Aiyao He, Shuai Xu, Hongming Zhang, Yanna Wang, Qingyang Zhang, Yajing Wang, Bo Xu

Beyond Browsing: API-Based Web Agents (2025.findings-acl)

Challenge:	Existing methods for function calling require expert effort and prompt engineering becomes inefficient.
Approach:	They propose a method that performs fine-grained, stepwise retrieval from a continually updated experience pool.
Outcome:	The proposed method achieves an average improvement of 6.1% on easy and 4.7% on hard questions.

Copied to clipboard

Yueqi Song, Frank F. Xu, Shuyan Zhou, Graham Neubig

Natural Language Interfaces

Challenge:	Existing web agents use browsers to facilitate human activities such as online shopping, online planning, and other work-related tasks.
Approach:	They propose to use web browsers as an interface to interact with online content through application programming interfaces (APIs) they propose to call APIs and use Hybrid Agents to perform online tasks.
Outcome:	The proposed agents outperform web Browsing Agents on a widely-used and realistic benchmark for web navigation tasks.

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)

Copied to clipboard

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov

API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs (2024.acl-long)

Challenge:	Language models have evolved from being research prototypes to commercialized products offered as web APIs.
Approach:	They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages.
Outcome:	The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results.

Copied to clipboard

Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis Lastras

InstructSafety: A Unified Framework for Building Multidimensional and Explainable Safety Detector through Instruction Tuning (2023.findings-emnlp)

Challenge:	Existing methods to train and test large language models that involve calls to tools and APIs are lacking.
Approach:	They propose a large corpora for training and systematic testing of tool-augmented LLMs.
Outcome:	The proposed datasets mimic real-world scenarios involving API-tasks and slot filling.

Copied to clipboard

Zhexin Zhang, Jiale Cheng, Hao Sun, Jiawen Deng, Minlie Huang

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion (2024.findings-emnlp)

Challenge:	Existing safety detection systems have limitations in terms of their versatility and interpretability.
Approach:	They introduce a safety detection framework that unifies 7 common sub-tasks into a uniform formulation and process 39 human-annotated datasets for instruction tuning.
Outcome:	The proposed framework unifies 7 common sub-tasks into a uniform formulation and then runs on 39 human-annotated datasets to fine-tune it.

Copied to clipboard

Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

MeaeQ: Mount Model Extraction Attacks with Efficient Queries (2023.emnlp-main)

Challenge:	Large Language Models (LLMs) are increasingly used for task completion in real-world situations.
Approach:	They propose a PowerPoint Task Completion-Robustness (PPTC-R) benchmark to measure LLMs’ robustness to the user PPT task instruction and software version (Powerpoint).
Outcome:	The proposed benchmark compares 3 closed-source and 4 open-source LLMs to the PowerPoint task instruction and software version (Powerpoint) .

Copied to clipboard

Chengwei Dai, Minxuan Lv, Kun Li, Wei Zhou

Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklists (2025.findings-acl)

Challenge:	Recent studies focus on limited-query budget settings and adopt random sampling or active learning-based sampling strategies on publicly available, unannotated data sources.
Approach:	They propose a model extraction attack with efficient Queries that uses a zero-shot sequence inference classifier to filter task-relevant data from a public text corpus instead of a problem domain-specific dataset.
Outcome:	The proposed method achieves higher similarity to the victim model than baselines while requiring fewer queries.

Copied to clipboard

Yue Cui, Liuyi Yao, Shuchang Tao, Weijie Shi, Yaliang Li, Bolin Ding, Xiaofang Zhou

AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction (2024.emnlp-main)

Challenge:	Large language models (LLMs) have advanced natural language processing, but their effectiveness is often hampered by parameter mis-filling during tool calling.
Approach:	They propose a hierarchical tool error checklist framework to diagnose and mitigate tool-calling errors without relying on extensive real-world interactions.
Outcome:	The proposed framework improves parameter-filling accuracy and tool-calling success rates compared to baseline methods.

Copied to clipboard

Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Pan, Kam-Fai Wong

Structured Data in NLP

Challenge:	Existing state-of-the-art Large Language Models (LLMs) still cannot perform well in this situation even with the help of in-context learning and finetuning.
Approach:	They propose a benchmark to evaluate LLMs’ ability to plan and execute multiple APIs from various sources in order to complete the user’s task.
Outcome:	The proposed benchmarks show that the existing state-of-the-art LLMs still cannot perform well in this situation even with in-context learning and finetuning.

ExploraCoder: Advancing Code Generation for Multiple Unseen APIs via Planning and Chained Exploration (2025.acl-long)

Copied to clipboard

Yunkun Wang, Yue Zhang, Zhen Qin, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, Shuiguang Deng

Programming Languages in NLP

Challenge:	Large language models face intrinsic limitations in coding with unseen APIs in training corpora.
Approach:	They propose a training-free framework that empowers LLMs to invoke multiple unseen APIs in code solution by planning a complex problem into several API invocation subtasks and experimenting with correct API usage at intermediate steps.
Outcome:	The proposed framework significantly improves performance for models lacking prior API knowledge, achieving 11.99% over retrieval-based approaches and 17.28% over pretraining-based methods in pass@10.

SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIs (2026.findings-acl)

Copied to clipboard

Wu Zhengxian, Juan Wen, Wanli Peng, Haowei Chang, Yinghan Zhou, Xue Yiming

API-Assisted Code Generation for Question Answering on Varied Table Structures (2023.emnlp-main)

Challenge:	Existing black-box instruction backdoors can detect poisoned inputs, but fail to recover correct outputs once the backdoor is activated.
Approach:	They propose a soft label mechanism and key-extraction-guided CoT-based defense against instruction backdoors in APIs (SLIP) they propose KCOT-based model to extract task-relevant keywords and phrases rather than only considering the single trigger or overall text semantics.
Outcome:	The proposed model reduces the average attack success rate to 25.13% and improves clean accuracy to 87.15% and outperforms state-of-the-art black-box defenses.

Copied to clipboard

Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, Daniel Fried

A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)

Challenge:	Existing approaches to table question answering have been limited to specific table structures.
Approach:	They propose a unified TableQA framework that uses Python as a querying language and few-shot prompting to translate NL questions into Python programs.
Outcome:	The proposed framework provides a unified representation for structured tables as multi-index Pandas data frames and uses Python as a powerful querying language to translate NL questions into Python programs.

Copied to clipboard

Alexandros Xenos, Themos Stafylakis, Ioannis Patras, Georgios Tzimiropoulos

VulAgent: Hypothesis-Validation Driven Multi-Agent Architecture for Vulnerability Detection (2026.findings-acl)

Challenge:	Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge.
Approach:	They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs .
Outcome:	The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets.

Copied to clipboard

Ziliang Wang, Ge Li, Jia Li, Hao Zhu, Zhi Jin

Challenge:	Recent reports indicate that software vulnerabilities caused by insecure coding practices remain a major security threat.
Approach:	They propose a multi-agent vulnerability detection framework based on hypothesis validation . they use multi-view analyzers to localize and localize security-sensitive operations .
Outcome:	The proposed framework reduces false positives and increases accuracy by 6.6 percentage points on PrimeVul and SVEN.

Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024.emnlp-main)

Copied to clipboard

Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, Fei Huang

Learning API Functionality from In-Context Demonstrations for Tool-based Agents (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) have revolutionized natural language processing with impressive capabilities, but they lack domain specificity, real-time information and face challenges in solving specialized problems.
Approach:	They propose a multi-LLM approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer.
Outcome:	The proposed model outperforms existing models by demonstrating its effectiveness and advantages in tool learning.

Copied to clipboard

Bhrij Patel, Ashish Jagmohan, Aditya Vempaty

Challenge:	Documentation is often missing, outdated, privatized, or inconsistent in tool-based agents.
Approach:	They propose a task that allows agents to learn API functionality directly from in-context demonstrations.
Outcome:	The proposed model improves the task success rate of an API-based agent by providing explicit function calls and natural language critiques.

SkyLLM: Cross-LLM-APIs Federation for Cost-effective Query Processing (2025.findings-acl)

Copied to clipboard

Heng Zhao, Yifei Zhu

Challenge:	Large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks, from text generation to complex problem-solving.
Approach:	They propose a system which federates multiple LLM APIs and dynamically assigns a non-empty subset of these APIs to each query prior to inference.
Outcome:	The proposed system can match the most accurate LLM with the lowest cost while cutting costs by 67.8%.

Sequential API Function Calling Using GraphQL Schema (2024.emnlp-main)

Copied to clipboard

Avirup Saha, Lakshmi Mandal, Balaji Ganesan, Sambit Ghosh, Renuka Sindhgatta, Carlos Eberhardt, Dan Debrunner, Sameep Mehta

GenesisFunc: Multi-Agent Data Generation for Accurate and Generalizable Function-Calling (2026.acl-long)

Challenge:	Function calling using Large Language Models (LLMs) is an active research area . however, sequential function calling using LLMs with interdependence between functions is still under-explored .
Approach:	They propose a dataset representing real-world REST API calls with variable mapping between functions using natural language utterances paired with function call sequences.
Outcome:	The proposed dataset represents real-world REST API calls with variable mapping between functions.

Copied to clipboard

Hao-Xiang Xu, Chong Deng, Jiaqing Liu, Wen Wang, Qian Chen, Lujia Bao, Xiangang Li, Zhen-Hua Ling

Challenge:	Large Language Models (LLMs) extend their capabilities through function-calling (FC) however, obtaining and annotating real function-called data is challenging, and synthetic data from existing pipelines suffers from unreliable APIs, limited tool scalability, insufficient diversity, and weak quality control.
Approach:	They propose a pipeline for generating FC training data using reliable tools and a multi-agent framework that supports a dialogue generation system that produces conversations spanning diverse scenarios.
Outcome:	The proposed pipeline outperforms open-source models in in-domain FC performance and out-of-domain generalization while reaching FC capabilities comparable to some of the latest API-based models.

RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning (2026.acl-long)

Copied to clipboard

Xiang Gao, Yuguang Yao, Qi Zhang, Kaiwen Dong, Avinash Baidya, Ruocheng Guo, Hilaf Hasson, Kamalika Das

NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)

Challenge:	Large language models (LLMs) struggle to use tools reliably in domain-specific settings.
Approach:	They propose a neuro-symbolic approach to adapt large language models to task-specific tools . they propose reusable rules that are distilled from failure traces and injected into the prompt .
Outcome:	Experiments show that the proposed approach outperforms prompting-based adaptation methods and complements finetuning.

Copied to clipboard

Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi

Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation (2026.acl-long)

Challenge:	Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them.
Approach:	They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call.
Outcome:	The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls.

Copied to clipboard

Huizi Cui, Huan Ma, Qilin Wang, Yuhang Gao, Changqing Zhang