Papers by Sachin Kumar
Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating text using Glauber dynamics are autoregressive, but they face a number of limitations. |
| Approach: | They propose a discrete diffusion-based generative model for text generation using Glauber dynamics from statistical physics and use pretrained causal/masked language models to improve the quality of the generated text. |
| Outcome: | The proposed model outperforms existing models on some common sense reasoning tasks and planning/search tasks. |
Reasoning Up the Instruction Ladder for Controllable Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Current models struggle to balance competing directives, causing conflicting instructions. |
| Approach: | They propose to reframe instruction hierarchy resolution as a reasoning task . they use a training dataset to enable this capability by transferring general reasoning capabilities to instruction prioritization . |
| Outcome: | The proposed method improves on safety-critical scenarios beyond the training distribution and jailbreaks. |
Machine Translation into Low-resource Language Varieties (2021.acl-short)
Copied to clipboard
| Challenge: | Current machine translation systems generate a "standard" target language, but many languages have multiple varieties that are different from the standard language. |
| Approach: | They propose a framework to rapidly adapt machine translation systems to generate different target varieties . they propose to use no parallel data to generate languages close to, but different from, the standard target language . |
| Outcome: | The proposed model improves on a system that generates Ukrainian and Belarusian in two languages with no parallel data. |
Topics to Avoid: Demoting Latent Confounds in Text Classification (D19-1)
Copied to clipboard
| Challenge: | Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well. |
| Approach: | They propose a method that represents latent topical confounds and a model which “unlearns” confounding features by predicting both the label of the input text and the confound. |
| Outcome: | The proposed model generalizes better and learns features indicative of the writing style rather than the content. |
TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm. |
| Approach: | They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models. |
| Outcome: | The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks. |
Steering off Course: Reliability Challenges in Steering Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior studies have evaluated a few steering methods for language models, leaving gaps in understanding their robustness. |
| Approach: | They examine three steering methods for language models to examine their reliability . they use function vectors, task vectors and DoLa to steer models toward desirable outputs . |
| Outcome: | The proposed methods show that they are not robust enough to handle large models with large parameters. |
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms across four benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode. |
| Approach: | They compare hypernetwork-based LoRA adaptation against carefully designed few-shot prompting in a controlled experiment . they find that few- shot prompting contributes +21.5% to performance and documentation contributes 0% . |
| Outcome: | The hypernetwork-based LoRA adaptation provides no measurable improvement over few-shot prompting alone. |
ClinicalTrialsHub: Bridging Registries and Literature for Comprehensive Clinical Trial Access (2026.eacl-demo)
Copied to clipboard
Jiwoo Park, Ruoqi Liu, Avani Jagdale, Andrew Srisuwananukorn, Jing Zhao, Lang Li, Ping Zhang, Sachin Kumar
| Challenge: | ClinicalTrialsHub consolidates clinical trial data from ClinicalTrial.gov and augments it by extracting and structuring trial-relevant information from PubMed. |
| Approach: | They propose a search-focused platform that consolidates PubMed data and extracts structured trial information. |
| Outcome: | ClinicalTrialsHub increases access to structured clinical trial data by 83.8% compared to ClinicalTrial.gov alone. |
GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing LLMs excel and often surpass human performance on benchmarks, but they are known to falter in simple tasks and under seemingly straightforward circumstances. |
| Approach: | They propose a benchmark to assess compositional and conditional reasoning within a flight booking task. |
| Outcome: | The proposed model outperforms existing models on the flight booking task with a 67% accuracy rate. |
Mitigating Societal Harms in Large Language Models (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | Recent studies have highlighted societal harms that can be caused by language generation models deployed in the wild. |
| Approach: | They propose to use a typology of technical approaches to mitigating harms of language generation models to provide an overview of potential social issues in language generation including toxicity, social biases, misinformation, factual inconsistency, and privacy violations. |
| Outcome: | The proposed typology addresses toxicity, biases, misinformation, factual inconsistency, and privacy violations in language generation models. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |
P3Sum: Preserving Author’s Perspective in News Summarization with Diffusion Language Models (2024.naacl-long)
Copied to clipboard
Yuhan Liu, Shangbin Feng, Xiaochuang Han, Vidhisha Balachandran, Chan Young Park, Sachin Kumar, Yulia Tsvetkov
| Challenge: | Existing summarization systems alter the political opinions and stances of news articles in more than 50% of summaries, misrepresenting the intent and perspectives of the authors. |
| Approach: | They propose a model-based summarization approach controlled by political perspective classifiers that preserves the political stance of a generated summary. |
| Outcome: | The proposed model outperforms state-of-the-art summarization systems and large language models by up to 13.7% in terms of success rate of stance preservation, with competitive performance on standard metrics of summarizing quality. |
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (2023.emnlp-main)
Copied to clipboard
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, Yulia Tsvetkov
| Challenge: | Language models have evolved from being research prototypes to commercialized products offered as web APIs. |
| Approach: | They conduct a systematic analysis of the cost and utility of OpenAI’s language model API on multilingual benchmarks in 22 typologically diverse languages. |
| Outcome: | The proposed language model API performs poorly on multiple languages and speakers of a large number of languages are overcharged while obtaining poorer results. |
Gradient-based Constrained Sampling from Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained language models are successful at generating fluent text but are notoriously hard to controllably sample from. |
| Approach: | They propose a sampling procedure that combines the log-likelihood of the language model with arbitrary constraints in a single energy function and then generates samples in . non-autoregressive manner. |
| Outcome: | The proposed method improves on text generation with soft and hard constraints and keyword-guided generation. |
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control (2023.acl-long)
Copied to clipboard
| Challenge: | Existing diffusion models for continuous-valued domains have not been adopted for text data. |
| Approach: | They propose a diffusion-based language model with two key design choices . semi-autoregressive model generates blocks of text and allows local context updates . they evaluate it on unconstrained text generation benchmarks . |
| Outcome: | The proposed model outperforms autoregressive models on unconstrained text generation benchmarks on uncontrolled text generation. |
Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical results show plug-and-play approach to reason about belief states of multiple characters in reading comprehension tasks is more precise and interpretable than previous approaches. |
| Approach: | They propose a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation. |
| Outcome: | The proposed algorithm improves theory of mind of off-the-shelf neural language models without supervision. |
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback (2025.acl-long)
Copied to clipboard
Lester James Validad Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
| Challenge: | Learning from human feedback has enabled the alignment of language models (LMs) with human preferences. |
| Approach: | They propose a Hybrid Preference routER that defers an annotation to either humans or LMs, achieving better annotation quality while reducing the cost of human-only annotation. |
| Outcome: | The proposed model achieves better annotation quality while reducing the cost of human-only annotation. |
A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing methods for word embeddings generate faster training with fewer learnable parameters. |
| Approach: | They propose a novel margin-based loss that uses only predicted and target embeddings . they argue that the loss is more consistent and interpretable than other margin--based losses . |
| Outcome: | The proposed model is more consistent and interpretable than other margin-based losses. |
FLEXITOKENS: Flexible Tokenization for Evolving Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Widely used subword tokenizers overfragment sequences in unseen domains, languages, and scripts . inefficient tokenizer models can cause overfragments in out-of-distribution domains if not trained properly . |
| Approach: | They propose a byte-level LM with learnable tokenizers to make tokenization adaptive . they propose 'flexitoken' which enables significantly greater flexibility during adaptation . |
| Outcome: | The proposed method significantly reduces token overfragmentation and improves on multilingual benchmarks and domains. |
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text generation evaluation metrics are lacking in robustness analysis. |
| Approach: | They propose to use stress tests to test for errors in text generation evaluation metrics . they find that BERTScore is confused by truncation errors in summarization . |
| Outcome: | The proposed stress tests show that they are insensitive to errors in open-ended generation, translation, and summarization. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in the capacity of large language models to generate human-like text have prompted a heated discourse around the risks of societal harms they introduce. |
| Approach: | They propose a taxonomy of interventions organized around the different phases where they can be adopted to mitigate harms. |
| Outcome: | The proposed methods are based on several prior works’ taxonomies of language model risks and provide an overview of strategies for detecting and ameliorating different kinds of risks/harms. |
Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new framework for sentence summarization is available that can be trained reference-free . a high-quality dataset of sentence-summary pairs with varying degrees of compression ratios is obtained . |
| Approach: | They propose a framework for sentence summarization that can be trained reference-free . they propose 'referee' that iteratively distills latent knowledge into better models . |
| Outcome: | The proposed framework outperforms existing models in the use of explicit examples from teacher models without compromising the quality of the summarization. |
David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies of diffusion-based language models have been conducted on a smaller scale. |
| Approach: | They propose to scale an autoregressive diffusion model from 0.4B to 13B parameters and propose techniques to improve its training and inference efficiency. |
| Outcome: | The proposed model is able to combine a large general-purpose diffusion model with smaller, but specialized and contextualized diffusion models at inference time. |