Papers by Varun Kumar
ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)
Copied to clipboard
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang
| Challenge: | Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area. |
| Approach: | They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. |
| Outcome: | The proposed model performs better on human annotators and on SOTA models with human annnotators. |
A Static Evaluation of Code Completion by Large Language Models (2023.acl-industry)
Copied to clipboard
Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta
| Challenge: | Large language models trained on code have shown great potential to increase productivity of software developers. |
| Approach: | They propose a static evaluation framework to quantify static errors in Python code completions by leveraging Abstract Syntax Trees. |
| Outcome: | The proposed framework is more efficient and applicable to code in the wild. |
CodeScout: Contextual Problem Statement Enhancement for Software Agents (2026.findings-acl)
Copied to clipboard
Manan Suri, Xiangci Li, Mehdi Shojaie, Songyang Han, Chao-Chun Hsu, Shweta Garg, Aniket Anand Deshmukh, Varun Kumar
| Challenge: | Current AI-powered code assistance tools struggle with ambiguous problem statements . failures on such ambiguously requests are highly correlated with longer trajectories . |
| Approach: | They propose a contextual query refinement approach that transforms ambiguous user requests into comprehensive, actionable problem statements through lightweight pre-exploration of the target codebase. |
| Outcome: | Empirical results show that CodeScout improves resolution rates with 27 additional issues resolved compared to baseline method. |
The Bull and the Bear: Summarizing Stock Market Discussions (2022.lrec-1)
Copied to clipboard
| Challenge: | a dataset of 7888 reddit posts and 400 posts is used to summarize stock market topics. |
| Approach: | They curate discussions on social media platforms and construct an abstractive summarization dataset. |
| Outcome: | The proposed dataset consists of 7888 Reddit posts and summaries for 400 posts . it is robustly evaluated and will be made publicly available . |
Investigating the Role and Impact of Disfluency on Summarization (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies have focused on disfluency detection and removal, with limited studies into its impact on downstream tasks. |
| Approach: | They propose to incorporate disfluency in summarization models to reduce the impact of replacement disfluencies on natural language processing tasks. |
| Outcome: | The proposed model improves on both public and real-life datasets and shows that it can handle disfluent data with up to 6.99-point degradation in Rouge-L score and replacement disfluencies have the highest negative impact. |
CODESTRUCT: Code Agents over Structured Action Spaces (2026.acl-long)
Copied to clipboard
| Challenge: | LLM-based code agents treat repositories as unstructured text, fail to produce valid patches . authors propose a structure-aware interface that exposes a codebase as a programmable action space . |
| Approach: | They propose to reframe the codebase as a structured action space where agents operate on named AST entities rather than text spans. |
| Outcome: | Evaluated on six LLMs, the proposed framework improves Pass@1 accuracy by 1.2-5.0% and reduces token consumption by 12-38%. |
ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Paraphrase generation is a long-standing task in natural language processing (NLP). |
| Approach: | They propose to generate large-scale syntactically diverse paraphrase datasets by abstract meaning representation back-translation. |
| Outcome: | The proposed dataset is syntactically more diverse than existing datasets while maintaining good semantic similarity. |
SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion (2026.acl-long)
Copied to clipboard
George Ma, Anurag Koul, Qi Chen, Yawen Wu, Sachit Kuhar, Yu Yu, Aritra Sengupta, Varun Kumar, Murali Krishna Ramanathan
| Challenge: | Large Language Models (LLMs) excel at code-related tasks but struggle in real software repositories. |
| Approach: | They propose a large-scale agent that injects repository context at inference time to improve both latency and code-generation quality by proactively exploring repository files during indexing and constructing speculative context. |
| Outcome: | Experiments show that SpecAgent achieves 9–11% relative performance gains compared to baselines while significantly reducing inference latency. |
Mirage: A Diagnostic Framework for Evaluating the Realism of Synthetic Contact Center Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | capturing the full realism of real agent–customer interactions remains a challenge . privacy constraints and data scarcity limit the availability of real conversations . |
| Approach: | They propose a diagnostic evaluation framework for synthetic dialogue generation . they benchmarked strategies guided by structured supervision on call attributes . |
| Outcome: | The proposed framework outperforms synthetic transcripts on quality assurance tasks . it shows that current generation strategies lack sentiment fidelity, disfluency modeling, behavioral variation and conversational realism . |
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022.acl-short)
Copied to clipboard
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, Aram Galstyan
| Challenge: | Recent natural language processing systems use large language models as the backbone . however, societal biases are encoded in these models and transferred to downstream applications . |
| Approach: | They propose to use two categories to measure fairness in natural language processing tasks . they find intrinsic and extrinsic metrics do not correlate in their original setting . |
| Outcome: | The proposed metrics do not correlate in their original setting, the authors show . they find that they are not accurate when correcting for metric misalignments and noise . |
Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Obtaining human annotation is expensive and time-consuming process. |
| Approach: | They propose a semi-supervised learning pipeline which leverages millions of unlabeled examples to improve natural language understanding tasks. |
| Outcome: | The proposed pipeline can be used to improve natural language understanding tasks. |
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal (2022.findings-acl)
Copied to clipboard
Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, Aram Galstyan
| Challenge: | Language models excel at generating coherent text, but can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions. |
| Approach: | They propose to modify teacher probabilities and augment the training set to learn a fair model during knowledge distillation by modifying teacher probability and augmenting the training sets. |
| Outcome: | The proposed approach reduces gender disparity in open-ended text generated from the distilled and finetuned models with only a minor compromise in utility. |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |
Resolving Ambiguities in Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
Ninareh Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Rahul Gupta
| Challenge: | ambiguities can lead to misinterpretation and miscommunication in natural language . resolving ambiguity is notoriously hard for machines . |
| Approach: | They propose a framework to disambiguate prompts given to generative models by soliciting clarifications from the end user. |
| Outcome: | The proposed framework generates more faithful images better aligned with user intention in the presence of ambiguities. |
Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to syntactically controlled paraphrase generation require annotated paraphrase pairs for training and are costly to extend to new domains. |
| Approach: | They propose to leverage Abstract Meaning Representations (AMR) to improve the performance of unsupervised syntactically controlled paraphrase generation. |
| Outcome: | The proposed model generates more accurate syntactically controlled paraphrases, both quantitatively and qualitatively, compared to the existing unsupervised approaches. |
Why Didn’t You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models (P19-1)
Copied to clipboard
| Challenge: | Informed prior-based methods provide better control than constraints, but constraints yield higher quality topics, but with less control. |
| Approach: | They propose to use constraints and informed prior-based methods to improve user control and topic coherence. |
| Outcome: | The proposed methods improve user control and topic coherence, while constraints yield higher quality topics, but with less control. |
Synthesizing Human Gaze Feedback for Improved NLP Performance (2023.eacl-main)
Copied to clipboard
| Challenge: | Prior work on eye tracking and NLP reveals that human scanpaths can aid in understanding and performance of NLP models. |
| Approach: | They propose a model for generating human scanpaths over text that approximates meaningful cognitive signals in human gaze patterns. |
| Outcome: | The proposed model can approximate meaningful cognitive signals in human gaze patterns. |
A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification (D19-61)
Copied to clipboard
| Challenge: | Recent studies have focused on the problem of generalizing from a few examples per category. |
| Approach: | They propose to use feature space data augmentation methods to improve intent classification performance in few-shot setting. |
| Outcome: | The proposed methods improve intent classification performance in few-shot setting beyond transfer learning approaches. |
Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations that focus on performance metrics miss a key factor: increased effectiveness due to additional compute. |
| Approach: | They propose to incorporate the compute budget into evaluations to provide a more informative comparison that takes into account both performance metrics and computational cost. |
| Outcome: | The proposed framework outperforms reasoning strategies when they use comparable compute resources. |
Planning-Aware Code Infilling via Horizon-Length Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to fill-in-the-middle (FIM) often fail to generate content that aligns well with the surrounding context. |
| Approach: | They propose a training objective that teaches models to predict the number of remaining middle tokens at each step. |
| Outcome: | The proposed training objective improves FIM performance by up to 24% on diverse benchmarks across file-level and repository-level. |