Papers by Mayank Agarwal
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi
| Challenge: | Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them. |
| Approach: | They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call. |
| Outcome: | The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls. |
How Good Are LLMs at Processing Tool Outputs? (2026.eacl-long)
Copied to clipboard
Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, Mayank Agarwal
| Challenge: | Real-world task automation tasks require large language models to call tools, which often return complex JSON responses. |
| Approach: | They evaluated 15 open and closed weight models using multiple prompting approaches to evaluate their tool response processing task and their ability to process structured (JSON) responses. |
| Outcome: | The proposed model can process structured (JSON) responses with 3% to 50% performance differences. |
Aligners: Decoupling LLMs and Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. |
| Approach: | They propose to decouple LLMs and alignment by training *aligner* models that can be used to align any LLM on an as-needed basis. |
| Outcome: | The proposed model can be used to align any LLM for a given criteria on an as-needed basis. |
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024.emnlp-industry)
Copied to clipboard
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G P Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi
| Challenge: | Existing research explores the use of Large Language Models (LLMs) as the backbone of agentic systems. |
| Approach: | They propose a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling that has better generalizability on multiple tasks across seven evaluation benchmarks. |
| Outcome: | The proposed model outperforms more than 15 other models on out-of-domain datasets and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL). |
Explain-then-translate: an analysis on improving program translation with self-generated explanations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using self-generated natural language explanations improves zero-shot performance by 12% on average. |
| Approach: | They propose to use self-generated natural language explanations as an intermediate step for code-to-code translation with language models. |
| Outcome: | The proposed approach improves zero-shot performance by 12% on average . the proposed approach is not evaluated on a broader set of languages including low-resource languages. |