Papers by Ali Payani
How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on tau-bench (2025.findings-emnlp)
Copied to clipboard
Venkatesh Mishra, Amir Saeidi, Satyam Raj, Mutsumi Nakamura, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral
| Challenge: | Recent advances in reasoning and planning capabilities of large language models have enabled their potential as autonomous agents capable of tool use in dynamic environments. |
| Approach: | They propose an input-reformulation multi-agent framework that reformulates user queries . |
| Outcome: | The proposed framework outperforms ReAct, Function Calling, and Self-Reflection in overall pass5 scores. |
FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments (2026.findings-acl)
Copied to clipboard
Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral
| Challenge: | Large Language Models are being increasingly deployed as decision-making core of autonomous agents . however, in conversational benchmarks, these agents fail due to the cascading effects of incorrect decision- making . |
| Approach: | They propose a framework that analyzes failure trajectories from baseline agents to identify most prevalent errors. |
| Outcome: | Experiments show that the framework improves performance over open-source LLMs . the framework can be used to build reliable, multi-turn tool-use agents . |
MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance (2025.findings-acl)
Copied to clipboard
| Challenge: | Multi-document reasoning is an area of increasing relevance given LLM capabilities in handling longer-context inputs, but few benchmarks exist to rigorously examine model behavior in this setting. |
| Approach: | They propose a new dataset for evaluating LLMs on the task of multi-document reasoning that uses condensed structured seed knowledge to modify it through LLM-assisted edits. |
| Outcome: | The proposed method generates document sets and QA examples on a multi-document reasoning task using a synthetic generation process. |
Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Logic-based approaches to reasoning have lost popularity due to limited scalability and coverage. |
| Approach: | They present a dataset of 28K sentence-level NL-FOL pairs from GPT4 and a LogicLLaMA2-7B/13B fine-tuned on MALLS for NL translation. |
| Outcome: | The proposed model can be used standalone or to correct previously generated rules by GPT3.5. |
When is Tree Search Useful for LLM Planning? It Depends on the Discriminator (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to build language agents that can plan efficiently and accurately have not met the needs of advanced planning methods to achieve such improvements. |
| Approach: | They propose to use iterative correction and tree search to solve multi-step problems in a language agent framework with three components: a generator, a discriminator, and a planning method. |
| Outcome: | The proposed methods improve performance on two tasks, text-to-SQL parsing and mathematical reasoning, while using discriminators with 90% accuracy. |
Large Language Models Can Learn Temporal Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Temporal reasoning (TR) is a fundamental ability of large language models (LLMs) however, there is neo-standard methods to perform TR, which are not suitable for large language model applications. |
| Approach: | They propose a framework to enhance temporal reasoning by using a latent representation, temporal graph (TG) instead of reasoning over the original context, they adopt a temporal representation that enhances TR learning. |
| Outcome: | The proposed framework improves the learning of language-based TR by incorporating a latent representation, temporal graph (TG) a synthetic dataset is constructed for fine-tuning LLMs on text-to-TG translation tasks and benchmarks. |
Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Vision-Language Models have demonstrated remarkable capabilities in processing both visual and textual information. |
| Approach: | They examine the challenge of alignment and misalignment in LVLMs through an explainability lens. |
| Outcome: | The findings highlight the need for standardized evaluation protocols and in-depth explainability studies. |
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate long one-sentence responses that are less effective because they overlook two crucial factors: intra-cluster similarity and inter-c cluster similarity. |
| Approach: | They propose a method that generalizes semantic entropy and uses token probabilities to quantify uncertainty in large language models. |
| Outcome: | The proposed method can be extended to white-box settings by incorporating token probabilities. |
Text-to-SQL Error Correction with Language Models of Code (2023.acl-short)
Copied to clipboard
Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa, Yu Su, Huan Sun
| Challenge: | Existing semantic parsers are not accurate enough for use in text-to-SQL parsing tasks. |
| Approach: | They propose to build clause-level edit models to correct SQL queries instead of token-level ones. |
| Outcome: | The proposed model improves the exact set match accuracy of different parsers by 2.4-6.5 and obtains up to 4.3 point absolute improvement over two strong baselines. |
Can LLMs Reason in the Wild with Programs? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown superior capability to solve reasoning problems with programs. |
| Approach: | They propose a task where an LLM is tasked to solve a reasoning problem of unknown type by identifying the sub-problems and their corresponding formalisms. |
| Outcome: | The proposed model can be fine tuned to achieve better performance on ambiguous and mixed scope problems. |
Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning (2025.findings-naacl)
Copied to clipboard
Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral
| Challenge: | Reasoning abilities of LLMs have been a key focus in recent years. |
| Approach: | They propose to use a college-level Multiple Choice Question-Answering task to identify LLM errors and evaluate their performance. |
| Outcome: | The proposed framework can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks. |
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in natural language understanding and generation, but controlling their behavior remains a challenge. |
| Approach: | They propose a supervised steering approach that operates in sparse, interpretable representation spaces. |
| Outcome: | The proposed approach achieves higher success rates with minimal degradation in generation quality compared to existing methods. |
Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Chain-of-Thought (CoT) methods struggle with consistency and verification in complex reasoning tasks. |
| Approach: | They propose a framework that integrates structured knowledge representation with learned planning. |
| Outcome: | The proposed framework outperforms existing Chain-of-Thought (CoT) methods on math reasoning, logical reasoning, and coding tasks. |