Papers by Kiran Kate
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi
| Challenge: | Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them. |
| Approach: | They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call. |
| Outcome: | The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls. |
How Good Are LLMs at Processing Tool Outputs? (2026.eacl-long)
Copied to clipboard
Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, Mayank Agarwal
| Challenge: | Real-world task automation tasks require large language models to call tools, which often return complex JSON responses. |
| Approach: | They evaluated 15 open and closed weight models using multiple prompting approaches to evaluate their tool response processing task and their ability to process structured (JSON) responses. |
| Outcome: | The proposed model can process structured (JSON) responses with 3% to 50% performance differences. |
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
Generating OpenAPI Specifications from Online API Documentation with Large Language Models (2025.acl-industry)
Copied to clipboard
Koren Lazar, Matan Vetzler, Kiran Kate, Jason Tsay, David Boaz, Himanshu Gupta, Avraham Shinnar, Rohith D Vallam, David Amid, Esther Goldbraich, Jim Laredo, Ateret Anaby Tavor
| Challenge: | API specifications are often presented as unstructured HTML pages, requiring external users to manually convert it into a structured format. |
| Approach: | They propose a framework that transforms long API documentation pages into consistent, machine-readable API specifications. |
| Outcome: | The proposed framework generalizes well across hundreds of APIs and produces valid OpenAPI specifications that encapsulate most of the information from the original documentation. |
Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling (2026.eacl-long)
Copied to clipboard
Benjamin Elder, Anupama Murthi, Jungkoo Kang, Ankita Rajaram Naik, Kiran Kate, Kinjal Basu, Danish Contractor
| Challenge: | Large language models rely on external tools and APIs to perform tasks specified in natural language. |
| Approach: | They propose a benchmark that transforms SQL queries from BIRD-SQL into executable API sequences. |
| Outcome: | The proposed benchmark evaluates 10 LLMs and 4 ReACT agents with low task completion rates and 50% task completion rate. |
Towards LLMs Robustness to Changes in Prompt Format Styles (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing prompt engineering techniques for adapting pre-trained LLMs to specific tasks are zero-shot prompting and few-shot supervised fine-tuning. |
| Approach: | They propose a technique for addressing prompt brittleness by diversifying the styles used in the prompt few-shot examples by using computer vision techniques. |
| Outcome: | Empirical results show that the proposed technique reduces style-induced prompt brittleness while improving overall performance across prompt variations and different datasets. |