Papers by Chaitanya Devaguptapu
Hybrid Graphs for Table-and-Text based Question Answering using LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for QA rely on fine-tuning and high-quality data, which is difficult to obtain. |
| Approach: | They propose a Hybrid Graph-based approach for Table-Text QA that leverages Large Language Models without fine-tuning. |
| Outcome: | The proposed approach improves Exact Match scores by 10% on Hybrid-QA and 5.4% on OTT-QA. |
Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Enterprise systems are crucial for enhancing productivity and strategic growth, but data is fragmented across multiple sources and access controls are complex. |
| Approach: | They propose a benchmark that simulates enterprise settings with 500 diverse tasks . they show that even the most capable models achieve only 41.8% task completion . |
| Outcome: | The proposed benchmark shows that even the most capable models achieve only 41.8% task completion. |
HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to answer multi-hop questions are query-agnostic and the extracted facts are ambiguous as they lack context. |
| Approach: | They propose to use a knowledge graph to extract query-relevant information from unstructured text. |
| Outcome: | The proposed method achieves performance improvements on two popular datasets. |
Evaluating Compound AI Systems through Behaviors, Not Benchmarks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Compound AI (CAI) systems combine LLMs with retrievers and tools to enable information-seeking applications in the real-world. |
| Approach: | They propose a behavior-driven evaluation framework that generates test specifications aligned with real usage contexts. |
| Outcome: | The proposed framework generates test specifications aligned with real usage contexts and transforms them into concrete test cases. |
Adaptive LLM Routing under Budget Constraints (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized natural language processing, but their varying capabilities and costs pose challenges in practical applications. |
| Approach: | They propose to use LLM routing as a contextual bandit problem to enable adaptive decision-making . they use offline human preference data to refine the embeddings through online bandit feedback . |
| Outcome: | The proposed model is based on a contextual bandit problem enabling adaptive decision-making . it learns from offline human preference data and refines it through online bandit feedback. |
Finding Needles in Images: Can Multi-modal LLMs Locate Fine Details? (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Multi-modal Large Language Models (MLLMs) have fundamentally transformed how machines understand and reason about visual information. |
| Approach: | They propose a benchmark to evaluate MLLMs' ability to locate and reason about fine-grained details within complex documents including newspapers, menus, and lecture images. |
| Outcome: | The proposed method improves on existing methods and shows that it can handle fine-grained document understanding tasks. |