Papers by Osbert Bastani
TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate incorrect responses based on made-up facts, which are called hallucinations. |
| Approach: | They propose a framework that combines Retrieval Augmented Generation with conformal prediction to provide the first end-to-end statistical correctness guarantee for RAG. |
| Outcome: | The proposed framework reduces prediction set size by 16.2% on average compared to an ablation. |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
LLM Program Optimization via Retrieval Augmented Search (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that large language models have difficulty with program optimization out-of-the-box. |
| Approach: | They propose a blackbox adaptation method that performs beam search over candidate optimizations by a training dataset. |
| Outcome: | The proposed method outperforms retrieval based on the source code in a number of ways. |
Few-Shot Novel Concept Learning for Semantic Parsing (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning algorithms typically require thousands of examples to learn novel concepts. |
| Approach: | They propose an algorithm for learning novel concepts by representing them as programs over existing concepts. |
| Outcome: | The proposed approach outperforms end-to-end neural semantic parsers in a few-shot novel concept learning setting. |
Counterfactual Explanations for Natural Language Interfaces (2022.acl-short)
Copied to clipboard
| Challenge: | Semantic parsing is a promising technique for enabling natural language interfaces, but human language can encode concepts that do not exist in the underlying system or are encoded using different language. |
| Approach: | They propose a novel approach for generating explanations of a natural language interface based on semantic parsing by providing a user with an utterance and a demonstration of their desired goal. |
| Outcome: | The proposed approach significantly improves user performance and generates explanations that more closely match the user’s intent compared to two ablations. |