Avirup Saha, Lakshmi Mandal, Balaji Ganesan, Sambit Ghosh, Renuka Sindhgatta, Carlos Eberhardt, Dan Debrunner, Sameep Mehta
| Challenge: | Function calling using Large Language Models (LLMs) is an active research area . however, sequential function calling using LLMs with interdependence between functions is still under-explored . |
| Approach: | They propose a dataset representing real-world REST API calls with variable mapping between functions using natural language utterances paired with function call sequences. |
| Outcome: | The proposed dataset represents real-world REST API calls with variable mapping between functions. |
Similar Papers
GraphQL Query Generation: A Large Training and Benchmarking Dataset (2024.emnlp-industry)
Copied to clipboard
Manish Kesarwani, Sambit Ghosh, Nitin Gupta, Shramona Chakraborty, Renuka Sindhgatta, Sameep Mehta, Carlos Eberhardt, Dan Debrunner
| Challenge: | GraphQL is a powerful query language for APIs, but crafting complex GraphqL queries can be challenging. |
| Approach: | a team of researchers has created a large-scale, cross-domain text-to-GraphQL query operation dataset . the dataset includes 10,940 training triples spanning 185 cross-source data stores and 957 test triples over 14 data stores. |
| Outcome: | The proposed dataset includes 10,940 training triples and 957 test triples over 14 data stores. |
Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significantly advanced autonomous agents, particularly in zero-shot tool usage, also known as function calling. |
| Approach: | They propose to integrate function descriptions into prompt formats and introduce a new Decision Token for conditional prompts. |
| Outcome: | The proposed decision token improves function-calling accuracy and relevance detection and a translation pipeline overcomes multilingual limitations. |
seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | **seqBench** allows systematic variation of several key complexity dimensions. |
| Approach: | They introduce a parametrized benchmark for probing sequential reasoning limits in Large Language Models through precise, multi-dimensional control over several key complexity dimensions. |
| Outcome: | The framework allows systematic variation of logical depth, backtracking requirements and noise ratio on state-of-the-art LLMs. |
Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling (2026.eacl-long)
Copied to clipboard
Benjamin Elder, Anupama Murthi, Jungkoo Kang, Ankita Rajaram Naik, Kiran Kate, Kinjal Basu, Danish Contractor
| Challenge: | Large language models rely on external tools and APIs to perform tasks specified in natural language. |
| Approach: | They propose a benchmark that transforms SQL queries from BIRD-SQL into executable API sequences. |
| Outcome: | The proposed benchmark evaluates 10 LLMs and 4 ReACT agents with low task completion rates and 50% task completion rate. |
Schema and Natural Language Aware In-Context Learning for Improved GraphQL Query Generation (2025.naacl-industry)
Copied to clipboard
| Challenge: | GraphQL is a flexible alternative to REST APIs, but generating complex queries remains challenging. |
| Approach: | They propose a framework that integrates GraphQL schemas with natural language inputs to improve query generation accuracy. |
| Outcome: | The proposed framework improves performance on a publicly available complex GraphQL dataset. |
On Evaluating LLMs’ Capabilities as Functional Approximators: A Bayesian Evaluation Framework (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the way we can formulate tasks in text-in-text-out format. |
| Approach: | They propose a new evaluation framework to comprehensively assess LLMs’ function modeling abilities by adopting a Bayesian perspective of function modeling. |
| Outcome: | The proposed evaluation framework enables LLMs to excel in utilizing prior knowledge to develop a strong understanding of the underlying function. |
GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated significant capabilities in processing and understanding text data. |
| Approach: | They propose a structure-based instruction-based method to enhance LLM performance on complex graph tasks. |
| Outcome: | The proposed framework outperforms open-source models on graph problem-solving, but the gap is narrowing. |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing state-of-the-art Large Language Models (LLMs) still cannot perform well in this situation even with the help of in-context learning and finetuning. |
| Approach: | They propose a benchmark to evaluate LLMs’ ability to plan and execute multiple APIs from various sources in order to complete the user’s task. |
| Outcome: | The proposed benchmarks show that the existing state-of-the-art LLMs still cannot perform well in this situation even with in-context learning and finetuning. |
Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to enable large language models to implement function calling are limited in their tool-use capabilities. |
| Approach: | They propose a controllable, target-driven approach to empower LLMs to operate external APIs only via prompts. |
| Outcome: | The proposed approach limits LLMs to executing simple tasks, e.g., API Selection and Argument Completion. |