Papers by Sumit Gulwani
One-to-many testing for code generation from (just) natural language (2024.findings-emnlp)
Copied to clipboard
| Challenge: | MBPP relies on test cases to generate the right signature, data contamination is a problem . adapted code generation benchmarks allow for the description to be underspecified with respect to syntactic properties of code. |
| Approach: | They propose a code generation benchmark that allows for the description to be underspecified with respect to syntactic properties of code. |
| Outcome: | The proposed model removes ambiguity about the semantics of the task from the descriptions and evaluates generated code on multiple sets of assertions to account for ambiguities in the syntax. |
TSTR: Target Similarity Tuning Meets the Real World (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs). |
| Approach: | They propose to use sentences from a larger language model to improve similarity between two NL inputs and associated code outputs. |
| Outcome: | The proposed model can be trained on a small number of training examples and is cost-effective. |
An empirical study of validating synthetic data for formula generation (2025.findings-naacl)
Copied to clipboard
Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen
| Challenge: | Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them. |
| Approach: | They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning. |
| Outcome: | The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them. |
STACKFEED: Structured Textual Actor-Critic Knowledge base editing with FEEDback (2025.emnlp-industry)
Copied to clipboard
Shashank Kirtania, Naman Gupta, Priyanshu Gupta, Sumit Gulwani, Arun Iyer, Suresh Parthasarathy Iyengar, Arjun Radhakrishna, Sriram K. Rajamani, Gustavo Soares
| Challenge: | Large Language Models (LLMs) often generate incorrect or outdated information, especially in low-resource settings or when dealing with private data. |
| Approach: | They propose a framework that iteratively refines the knowledge base based on expert feedback . they define a ReACT actor agent on each document to perform structured edits . |
| Outcome: | The proposed framework improves the quality and performance of the RAG system on low-resource programming problems, modified Python packages, and factual question-answering tasks. |
TEN: Table Explicitization, Neurosymbolically (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for extracting tabular data from semistructured text are error-prone and costly. |
| Approach: | They propose a neurosymbolic approach to extract tabular data from semistructured text . TEN is a triadic feedback loop that iteratively refines table hypotheses . |
| Outcome: | The proposed approach outperforms neural baselines in exact match accuracy and lower hallucination rates. |
TeCoFeS: Text Column Featurization using Semantic Analysis (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for topic modeling and feature extraction are based on syntactic features and overlook the semantics. |
| Approach: | They propose a semantic text column featurization problem that extracts a small sample smartly using an LLM to label only the sample and then extends that labeling to the whole column using text embeddings. |
| Outcome: | The proposed approach performs better than baselines and naive use of LLMs. |
RAR: Retrieval-augmented retrieval for code generation in low resource languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Either examples or documentation are commonly used for improved code generation. |
| Approach: | They propose retrieval augmented retrieval as a two-step method for selecting relevant examples and documentation. |
| Outcome: | The proposed method outperforms example and grammar retrieval on low-resource languages . it also outperformed two-step retrieval when used independently . |
CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text. |
| Approach: | They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language. |
| Outcome: | The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models. |
MetaReflection: Learning Instructions for Language Agents using Past Reflections (2024.emnlp-main)
Copied to clipboard
Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, Sherry Shi
| Challenge: | Large Language Models (LLMs) have gained popularity due to their ability to generate human-like text and solve complex tasks. |
| Approach: | They propose an offline reinforcement learning technique that augments a semantic memory based on experiential learnings from past trials. |
| Outcome: | The proposed technique boosts Language agents’ performance by 4 % to 16.82 % over the raw GPT-4 baseline and performs on par with existing state-of-the-art prompt optimization techniques while requiring fewer LLM calls. |