Papers by Danish Danish
Prompting with Pseudo-Code Instructions (2023.emnlp-main)
Copied to clipboard
| Challenge: | ambiguity in natural language can hinder performance of large language models. |
| Approach: | They manually create a dataset of pseudo-code prompts for 132 different classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. |
| Outcome: | The pseudo-code prompts improve the performance of two LLM families, BLOOM and CodeGen. |
Downstream Trade-offs of a Family of Text Watermarks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can generate humanlike responses to a variety of requests like writing emails, translating or summarizing content. |
| Approach: | They evaluate the performance of large language models (LLMs) watermarked using three different strategies over a diverse suite of tasks including those cast as k-class classification (CLS), multiple choice question answering (MCQ), short-form generation (e.g., open-ended question answering) and long-form generator (eg. translation) |
| Outcome: | The proposed models can cause significant drops in their effectiveness across a variety of tasks including CLS, MCQ, short-form generation and translation tasks. |
Exemplar Encoder-Decoder for Neural Conversation Generation (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to generate conversational systems suffer from lack of diversity in responses and generation of short, repetitive and uninteresting responses. |
| Approach: | They propose a novel conversation model that uses similar examples from training data to generate responses. |
| Outcome: | The proposed model outperforms state-of-the-art sequence to sequence learning on several evaluation metrics on two large data sets. |
Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers (2020.lrec-1)
Copied to clipboard
Basil Abraham, Danish Goel, Divya Siddarth, Kalika Bali, Manu Chopra, Monojit Choudhury, Pratik Joshi, Preethi Jyoti, Sunayana Sitaram, Vivek Seshadri
| Challenge: | Existing platforms collect labelled speech data from urban speakers whose dialects are often very different from low-income users. |
| Approach: | They propose to collect labelled speech data directly from low-income workers . they collect 109 hours of data from 36 participants in the Marathi language . |
| Outcome: | The proposed approach can provide valuable supplemental earning opportunities to low-income rural and urban workers. |
All That Glitters is Not Novel: Plagiarism in AI Generated Research (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies claim autonomous research agents can generate novel research ideas. |
| Approach: | They ask experts to evaluate whether existing work is similar to new ones . they find 24% of the 50 evaluated documents to be either paraphrased or significantly borrowed . |
| Outcome: | The authors find that 24% of the 50 evaluated research documents are either paraphrased, or significantly borrowed from existing work. |
Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations (2025.findings-naacl)
Copied to clipboard
| Challenge: | a large body of work examines language models for biases concerning gender, race, occupation and religion . however, the impact of the encoded geographical knowledge on real-world applications has not been documented . |
| Approach: | They examine large language models for two common scenarios that require geographical knowledge: travel recommendations and geo-anchored story generation. |
| Outcome: | The results show that the language models are biased against poorer countries and poorer socioeconomic conditions. |
Knowledge Graph Guided Evaluation of Abstention Techniques (2025.naacl-long)
Copied to clipboard
| Challenge: | Several prior studies test the safety promises of language models based on their effectiveness in blocking malicious requests. |
| Approach: | They create a benchmark based on benign concepts and ground them in a knowledge graph to evaluate abstention techniques. |
| Outcome: | The proposed framework causes models to abstain with over 80% abstention rates, but not as effective for descendants of the target concepts, where abstraction rates drop by 19%. |
Simple and Effective Semi-Supervised Question Answering (N18-2)
Copied to clipboard
| Challenge: | Existing deep learning systems for extractive Question Answering are limited and expensive to construct. |
| Approach: | They propose a semi-supervised QA system where end user specifies a set of documents and only a few labelled examples. |
| Outcome: | The proposed system achieves 50% F1 score on SQuAD and TriviaQA with very little labeled data. |
Language Diversity: Visible to Humans, Exploitable by Machines (2022.acl-demo)
Copied to clipboard
Gábor Bella, Erdenebileg Byambadorj, Yamini Chandrashekar, Khuyagbaatar Batsuren, Danish Cheema, Fausto Giunchiglia
| Challenge: | Universal Knowledge Core (UKC) is a large multilingual lexical database with a focus on language diversity and covering over two thousand languages. |
| Approach: | Universal Knowledge Core is a large multilingual lexical database with a focus on language diversity and covering over two thousand languages. |
| Outcome: | the database lets users explore millions of individual words and their meanings, but also phenomena of cross-lingual convergence and divergence, such as shared interlingual meanings and lexicon similarities. |
Neural Conversational QA: Learning to Reason vs Exploiting Patterns (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural Conversational QA tasks such as ShARC require systems to answer questions based on the contents of a given passage. |
| Approach: | They propose to modify a data-set with fewer spurious patterns to exploit them . they also propose to build a heuristic-based program to exploit spurious clues . |
| Outcome: | The proposed program exploits spurious patterns in the ShARC dataset, compared to neural models. |
Evaluating Reasoning Models for Queries with Presuppositions (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work notes that large language models fail to challenge erroneous assumptions and can reinforce users’ misinformed opinions. |
| Approach: | They construct queries with varying degrees of presuppositions spanning health, science, and general knowledge and evaluate several widely-deployed models. |
| Outcome: | The proposed models achieve higher accuracy but fail to challenge a large fraction of false presuppositions. |
FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes (2025.acl-long)
Copied to clipboard
Janki Atul Nawale, Mohammed Safi Ur Rahman Khan, Janani D, Mansi Gupta, Danish Pruthi, Mitesh M Khapra
| Challenge: | Existing studies on fairness of LLMs are largely Western-focused, making them inadequate for culturally diverse countries such as India. |
| Approach: | They propose a benchmark to evaluate fairness of LLMs across 85 identity groups . they consult domain experts to curate over 1,800 socio-cultural topics . |
| Outcome: | The benchmark evaluates LLMs across 85 identities across 85 castes, religions, regions, and tribes. |
Weakly- and Semi-supervised Evidence Extraction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to combine evidence annotations with document labels are limited to a minority of training examples. |
| Approach: | They propose to combine evidence annotations with abundant document labels for evidence extraction task. |
| Outcome: | The proposed method outperforms baselines on two classification tasks with evidence annotations. |
NeuSpell: A Neural Spelling Correction Toolkit (2020.emnlp-demos)
Copied to clipboard
| Challenge: | a new spelling correction toolkit is available for free. |
| Approach: | They propose an open-source toolkit for spelling correction in English . they train neural models using spelling errors in context and using richer contextual representations. |
| Outcome: | The proposed spell-checker improves accuracy on synthetic examples and richer representations of the context. |
Agent Assist through Conversation Analysis (2020.emnlp-demos)
Copied to clipboard
Kshitij Fadnis, Nathaniel Mills, Jatin Ganhotra, Haggai Roitman, Gaurav Pandey, Doron Cohen, Yosi Mass, Shai Erera, Chulaka Gunasekara, Danish Contractor, Siva Patel, Q. Vera Liao, Sachindra Joshi, Luis Lastras, David Konopnicki
| Challenge: | Using conversational approach to information retrieval for agent assistance, customer support agents are a critical part of an organization's customer support team. |
| Approach: | They propose a conversational approach to information retrieval for agent assistance that monitors an evolving conversation and recommends both responses and URLs of documents. |
| Outcome: | The proposed system monitors an evolving conversation and recommends both responses and URLs of documents the agent can use in replies to their client. |
Combating Adversarial Misspellings with Robust Word Recognition (P19-1)
Copied to clipboard
| Challenge: | a sub-field of word recognition models is emerging to combat adversarial spelling mistakes . imperceptible attacks can cause models to misclassify examples, but training robust models remains a challenge . |
| Approach: | They propose to place a word recognition model in front of a downstream classifier to combat adversarial spelling mistakes. |
| Outcome: | The proposed model outperforms adversarial training and off-the-shelf spell checkers in a word recognition task. |
Multi-Level Memory for Task Oriented Dialogs (N19-1)
Copied to clipboard
| Challenge: | Recent task oriented dialog systems use memory architectures to incorporate external knowledge in their dialogs. |
| Approach: | They propose a novel multi-level memory architecture that separates dialog context and knowledge base results . they use cells for each query and their corresponding results to address queries . |
| Outcome: | The proposed architecture outperforms current state-of-the-art models on three publicly available data sets. |
Joint Reasoning on Hybrid-knowledge sources for Task-Oriented Dialog (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing systems for task oriented dialog use knowledge present only in structured knowledge sources to generate responses. |
| Approach: | They propose a model that assumes that information is always present in a structured knowledge base . they also refine the model to take into account the fact that it can fuse information from structured and unstructured knowledge sources. |
| Outcome: | The proposed model is robust to perturbations to knowledge modality and can fuse information from structured and unstructured knowledge to generate responses. |
Do Context-Aware Translation Models Pay the Right Attention? (2021.acl-long)
Copied to clipboard
| Challenge: | Context-aware machine translation models fail to leverage contextual information to resolve ambiguous words and pronouns. |
| Approach: | They propose a new dataset that includes supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. |
| Outcome: | The proposed model can automatically disambiguate pronouns and polysemous words when they are not in the same context. |
Model-tuning Via Prompts Makes NLP Models Adversarially Robust (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models are typically adapted to downstream tasks by appending a randomly initialized multilayer perceptron to their topmost representation layer and fine-tuning the entire model on a downstream task. |
| Approach: | They propose to append a multilayer perceptron to a CLS token and fine-tune the entire model on a downstream task. |
| Outcome: | The proposed model-tuning via prompts outperforms adversarial training-based state-of-art defenses by 3.5% and improves against adversarials by 8% over standard methods. |
Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Popular dialog datasets such as MultiWOZ are created by providing crowd workers with instructions that describe the task to be accomplished. |
| Approach: | They propose a data creation strategy that uses a pre-trained language model to simulate the interaction between crowd workers by creating a user bot and an agent bot. |
| Outcome: | The proposed data creation strategy improves on two publicly available datasets using a pre-trained language model and a smaller percentage of actual crowd-generated conversations and their corresponding instructions. |
Learning to Deceive with Attention-Based Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | Attention mechanisms are ubiquitous components in neural network architectures and are often claimed to confer interpretability. |
| Approach: | They propose a method for training models to produce deceptive attention masks by combining weights assigned to designated impermissible tokens with a weighted sum. |
| Outcome: | The proposed method reduces the weight assigned to designated impermissible tokens while still using them across multiple models and tasks. |
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (N19-4)
Copied to clipboard
| Challenge: | Unlike machine translation, natural language outputs are nuanced and there are no clear yes/no distinctions about whether they are correct or not. |
| Approach: | They describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. |
| Outcome: | The compare-mt tool is an open-source pure-python package that has already proven useful to generate analyses that have been used in our papers. |
Learning the Legibility of Visual Text Perturbations (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing adversarial attacks in NLP perturb text to produce visually similar strings ('ergo', 'rgo') which are legible to humans but degrade model performance. |
| Approach: | They use a human-annotated dataset comprising the legibility of visually perturbed text to build models that predict the legible inputs and rank them based on their legibility. |
| Outcome: | The proposed models achieve an F score of 0.91 and an accuracy of 0.86 in predicting which of two perturbations is more legible. |
Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for dialog response retrieval embed the context-response pairs as points in the embedding space. |
| Approach: | They propose a scalable model that can learn complex relationships between context-response pairs . they train the models by optimizing the Kullback-Leibler divergence between the distributions induced by context-responders in the training data. |
| Outcome: | The proposed model performs better than other embedding-based approaches on public conversation data. |
Geographical Erasure in Language Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models encode vast amounts of world knowledge but are at risk of inordinately capturing information about dominant groups. |
| Approach: | They propose to operationalise a form of geographical erasure wherein language models underpredict certain countries. |
| Outcome: | The proposed model underpredicts certain countries by a factor 3 . the model is based on large datasets and is able to mitigate the effects . |
Why and when should you pool? Analyzing Pooling in Recurrent Architectures (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Various pooling techniques have been shown to improve performance of RNNs on text classification tasks. |
| Approach: | They propose a pooling-based variant that captures interactions among predictive tokens in a sentence. |
| Outcome: | The proposed pooling architecture outperforms non-pooling models on sequence classification tasks. |
Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling (2026.eacl-long)
Copied to clipboard
Benjamin Elder, Anupama Murthi, Jungkoo Kang, Ankita Rajaram Naik, Kiran Kate, Kinjal Basu, Danish Contractor
| Challenge: | Large language models rely on external tools and APIs to perform tasks specified in natural language. |
| Approach: | They propose a benchmark that transforms SQL queries from BIRD-SQL into executable API sequences. |
| Outcome: | The proposed benchmark evaluates 10 LLMs and 4 ReACT agents with low task completion rates and 50% task completion rate. |
Revisiting the Robustness of Watermarking to Paraphrasing Attacks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent watermarking techniques slightly modify the output probabilities of LMs to embed a signal in the generated output that can later be detected. |
| Approach: | They propose to use a model to produce a watermarking signal that is invariant to semantically-similar inputs to undo the effects of watermarks. |
| Outcome: | The proposed method undoes the effects of watermarking and dramatically improves the effectiveness of paraphrasing attacks with limited access to model generations. |
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)
Copied to clipboard
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
| Challenge: | Existing methods to explain predictions by highlighting salient features are often unstated. |
| Approach: | They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. |
| Outcome: | The proposed framework allows principled, automatic, model-agnostic evaluation of attributions. |
Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering (2026.eacl-long)
Copied to clipboard
| Challenge: | In many real-world applications, users rely on natural language instructions to guide large language models (LLMs) However, LLMs do not attend to these instructions reliably, and users lack simple mechanisms to emphasize their importance beyond modifying prompt wording or structure. |
| Approach: | They propose an inference-time method that enables users to emphasize specific parts of their prompt by steering the model’s attention toward them, aligning the model's perceived importance of different tokens with user intent. |
| Outcome: | The proposed method improves instruction following across tasks involving multiple instructions and generalizes across models of varying scales. |
Semi-Structured Object Sequence Encoders (2023.findings-emnlp)
Copied to clipboard
Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara, Siva Patel, Hui Wan, Tejas Dhamecha, Danish Contractor, Marina Danilevsky
| Challenge: | Semi-structured object sequences are often represented as a sequence of key-value pairs over time . authors propose a two-part approach that takes each key independently and encodes a representation of its values over time. |
| Approach: | They propose a two-part approach that first considers each key independently and encodes a representation of its values over time. |
| Outcome: | The proposed approach outperforms existing methods on multiple prediction tasks using real-world data. |
Silencing Empowerment, Allowing Bigotry: Auditing the Moderation of Hate Speech on Twitch (2025.acl-long)
Copied to clipboard
| Challenge: | To meet the demands of content moderation, online platforms have resorted to automated systems. |
| Approach: | They conduct an audit of Twitch’s automated moderation tool (AutoMod) to investigate its effectiveness in flagging hateful content. |
| Outcome: | The automated moderation tool (AutoMod) is used to filter hateful content on Twitch and send 107,000 comments from 4 datasets. |
Goodhart’s Law Applies to NLP’s Explanation Benchmarks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Popular methods for "explaining" the outputs of natural language processing (NLP) models operate by highlighting a subset of input tokens that ought, in some sense, to be salient. |
| Approach: | They propose to inflate a model’s comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution inputs. |
| Outcome: | The proposed metrics exploit the tendency for extracted explanations and complements to be “out-of-support” relative to each other and in-distribution inputs. |
Evaluating Large Language Models for Health-related Queries with Presuppositions (2024.findings-acl)
Copied to clipboard
| Challenge: | a large number of health-related queries require factually accurate answers . however, the lack of accurate answers may cause real-world harm . |
| Approach: | They evaluate the factual accuracy and consistency of large language models using a dataset consisting of health-related queries with varying degrees of presuppositions. |
| Outcome: | The proposed model responses agree with 23-32% of existing false claims and 49-55% with novel fabricated claims. |