Papers by Mukul Singh
One-to-many testing for code generation from (just) natural language (2024.findings-emnlp)
Copied to clipboard
| Challenge: | MBPP relies on test cases to generate the right signature, data contamination is a problem . adapted code generation benchmarks allow for the description to be underspecified with respect to syntactic properties of code. |
| Approach: | They propose a code generation benchmark that allows for the description to be underspecified with respect to syntactic properties of code. |
| Outcome: | The proposed model removes ambiguity about the semantics of the task from the descriptions and evaluates generated code on multiple sets of assertions to account for ambiguities in the syntax. |
TSTR: Target Similarity Tuning Meets the Real World (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs). |
| Approach: | They propose to use sentences from a larger language model to improve similarity between two NL inputs and associated code outputs. |
| Outcome: | The proposed model can be trained on a small number of training examples and is cost-effective. |
An empirical study of validating synthetic data for formula generation (2025.findings-naacl)
Copied to clipboard
Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen
| Challenge: | Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them. |
| Approach: | They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning. |
| Outcome: | The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them. |
InstructExcel: A Benchmark for Natural Language Instruction in Excel (2023.findings-emnlp)
Copied to clipboard
Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri
| Challenge: | Large Language Models (LLMs) can solve increasingly complex NLP tasks such as Excel specific tasks. |
| Approach: | They propose a large-scale benchmark to test whether Large Language Models can generate code that solves Excel specific tasks provided via natural language user instructions. |
| Outcome: | The proposed model outperforms existing models and provides a hard benchmark for state of the art models like GPT-4. |
Agentic Context Strategies for Multi-Format Document Understanding: When Should Language Models Use Tools? (2026.acl-industry)
Copied to clipboard
| Challenge: | Large language models face fundamental trade-offs when processing long documents . full context is expensive and may exceed limits, while RAG risks missing relevant information. |
| Approach: | They evaluate four context strategies across three document formats to evaluate their results . they find agentic tool-augmented approaches outperform passive strategies . |
| Outcome: | The agentic tool-augmented approaches outperform passive strategies on three document formats. |
TeCoFeS: Text Column Featurization using Semantic Analysis (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for topic modeling and feature extraction are based on syntactic features and overlook the semantics. |
| Approach: | They propose a semantic text column featurization problem that extracts a small sample smartly using an LLM to label only the sample and then extends that labeling to the whole column using text embeddings. |
| Outcome: | The proposed approach performs better than baselines and naive use of LLMs. |
RAR: Retrieval-augmented retrieval for code generation in low resource languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Either examples or documentation are commonly used for improved code generation. |
| Approach: | They propose retrieval augmented retrieval as a two-step method for selecting relevant examples and documentation. |
| Outcome: | The proposed method outperforms example and grammar retrieval on low-resource languages . it also outperformed two-step retrieval when used independently . |
CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text. |
| Approach: | They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language. |
| Outcome: | The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models. |