Papers by Ameya Godbole
Analysis of Plan-based Retrieval for Grounded Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large, parametric language models (LLMs) produce fluent text for many applications . hallucinations are generation of text that is factually correct and semantically plausible . |
| Approach: | They propose to use learning-tuned LLMs to infuse models with retrieval mechanisms to reduce hallucinations. |
| Outcome: | The proposed approach reduces the frequency of hallucinations by reducing the coverage of relevant facts and generating more informative responses while providing higher attribution rates. |
SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples (2023.emnlp-main)
Copied to clipboard
| Challenge: | Manually collecting challenging negative examples can be costly and domain-specific. |
| Approach: | They propose a method that synthesizes negative examples from only positive ones using a mask infilling model. |
| Outcome: | The proposed method closes 69.6% of the performance gap on a dataset where half of the evaluation examples are unanswerable compared to a model trained on SQuAD 2.0. |
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (D19-58)
Copied to clipboard
Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, Zhiyu Gong, Abhishek Singhal, Mo Yu, Xiaoxiao Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, Andrew McCallum
| Challenge: | Multi-hop question answering (QA) requires an information retrieval system that can find multiple supporting evidence needed to answer the question. |
| Approach: | They propose a technique that uses information of entities present in the initial retrieved evidence to learn to ‘hop’ onto other relevant evidence. |
| Outcome: | The proposed method boosts retrieval performance on a multi-hop question answering dataset with 5 million Wikipedia paragraphs and a model without training increases its performance by 10.59 F1. |
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021.emnlp-main)
Copied to clipboard
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, Andrew McCallum
| Challenge: | Using human-labeled examples, case-based reasoning can solve complex problems from scratch . case-Based reasoning is a paradigm that is used to solve complex problem . |
| Approach: | They propose a neuro-symbolic CBR approach for question answering over large knowledge bases. |
| Outcome: | The proposed approach outperforms the current state of the art on a CWQ dataset by 11% on accuracy. |
Benchmarking Long-tail Generalization with Likelihood Splits (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to test out of distribution generalization have their own strengths and weaknesses. |
| Approach: | They propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. |
| Outcome: | The proposed approach can be customized to construct meaningful splits for a wide range of tasks. |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability (2025.emnlp-demos)
Copied to clipboard
Mohammad Aflah Khan, Ameya Godbole, Johnny Wei, Ryan Yixiang Wang, James Flemings, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
| Challenge: | Existing workflows for pretraining large language models are cumbersome, fragmented and inaccessible. |
| Approach: | They propose an open-source library for editing, inspection, and analysis of large language model datasets. |
| Outcome: | TokenSmith is an open-source library for editing, inspection, and analysis of large language model datasets. |
Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for learning non-parametric representations of entities and relations are based on tensor factorization or sophisticated neural approaches. |
| Approach: | They propose a case-based reasoning system that retrieves ‘cases’ that are similar to the given problem and then stores them in its parameters. |
| Outcome: | The proposed model outperforms state-of-the-art methods on several benchmark datasets and is non-parametric and grows dynamically as new entities and relations arrive in the KB. |
Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference (D19-53)
Copied to clipboard
| Challenge: | EMNLP 2019 shared task on 'Multi-hop Inference Explanation Regeneration' identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Approach: | They propose a system that identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Outcome: | The proposed system outperforms the second best system by 14.95 points on the mean average precision (MAP) metric. |