Papers by Harsh Jhamtani
Towards Robust Evaluation of Unlearning in LLMs via Data Transformations (2024.findings-emnlp)
Copied to clipboard
Abhinav Joshi, Shaswati Saha, Divyaksh Shukla, Sriram Vema, Harsh Jhamtani, Manas Gaur, Ashutosh Modi
| Challenge: | Large Language Models (LLMs) have shown to be a great success in a wide range of applications ranging from regular NLP-based use cases to AI agents. |
| Approach: | They examine the robustness of existing MUL techniques for their ability to enable leakage-proof forgetting in LLMs. |
| Outcome: | The proposed methods can be used to enable leakage-proof forgetting in LLMs. |
Learning Rhyming Constraints using Structured Adversaries (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to text generation fail to capture higher-level structure in text, for example, rhyming patterns. |
| Approach: | They propose a method that uses a structured discriminator to learn rhyming constraints from poetry . the discriminator compares two English poetry datasets based on a learned similarity matrix . |
| Outcome: | The proposed method can learn rhyming patterns in English poetry without explicit phonetic information. |
Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)
Copied to clipboard
Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas
| Challenge: | In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Approach: | They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Outcome: | The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting. |
Learning to Generate Move-by-Move Commentary for Chess Games from Large-Scale Social Forum Data (P18-1)
Copied to clipboard
| Challenge: | Using a large-scale chess commentary dataset, we generate a set of comments for individual moves in a game. |
| Approach: | They propose a large-scale chess commentary dataset and a method to generate commentary for individual moves in a chessian game. |
| Outcome: | The proposed method is rated similar to ground truth commentary texts in terms of correctness and fluency. |
The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding (2023.findings-acl)
Copied to clipboard
Hao Fang, Anusha Balakrishnan, Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, Dan Klein
| Challenge: | In a task-oriented dialogue system, response generation is a conditional language model, but effective dialogue agents must balance fluent generation with stricter constraints. |
| Approach: | They propose a rule-based content selection model that transduces a dialogue agent’s actions and their results into context-free grammars representing the space of contextually acceptable responses. |
| Outcome: | The proposed architecture outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness. |
Interpreting User Requests in the Context of Natural Language Standing Instructions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to LLM-based dialogue modeling provide additional context for users to make requests. |
| Approach: | They propose an approach to LLM-based dialogue modeling where persistent user constraints and preferences are provided as additional context for such interfaces. |
| Outcome: | The proposed model achieves a maximum of 46% exact match on the prediction of 2.4K English dialogues with a language-to-program dataset. |
LLM Agents for Coordinating Multi-User Information Gathering (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) are becoming a crucial building block in developing automated agents that can assist human users with complex tasks. |
| Approach: | They introduce PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. |
| Outcome: | The proposed benchmarks are adapted from existing benchmarks for database question answering and multi-document summarization. |
Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for target-guided response generation are inconsistent with human judgement ratings. |
| Approach: | They propose a technique that finds a bridging path between the source and target and uses it to generate transition responses. |
| Outcome: | The proposed technique outperforms baselines on target-guided response generation task. |
Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior work has shown that having multiple valid references is important for automated evaluations. |
| Approach: | They propose a technique for automatically expanding a human generated reference to a set of candidate references. |
| Outcome: | The proposed method improves correlations between human-generated metrics and human ratings of system outputs. |
Narrative Text Generation with a Latent Discrete Plan (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on story generation has focused on generating consistent stories via story outlines using keywords or key phrases. |
| Approach: | They propose a deep latent variable model that first samples a sequence of anchor words, one per sentence in the story, as part of its generative process. |
| Outcome: | The proposed model gets favorable scores when evaluated on perplexity, diversity, and control of story via discrete plan. |
Unsupervised Enrichment of Persona-grounded Dialog with Background Stories (2021.acl-short)
Copied to clipboard
| Challenge: | Existing dialog models do not contain such narratives, so we propose a gradient-based rewriting technique to enrich dialog personas with relevant background events. |
| Approach: | They propose to use existing dialog datasets to enrich dialog responses with 'background stories' based on a gradient-based rewriting technique which encourages the generated response to be fluent with the dialog history, minimally different from the retrieved story, and consistent with the original persona. |
| Outcome: | The proposed method generates responses that are more diverse and human-like compared to outputs from existing dialog models. |
Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | despite rapid progress in multihop question-answering, models still have trouble explaining why an answer is correct. |
| Approach: | They propose three explanation datasets in which explanations from corpus facts are annotated . they first annotate multiple candidate explanations for each answer, then use crowd-sourcing perturbations to test generalization . |
| Outcome: | The proposed datasets improve explanation quality but still behind the upper bound . the proposed dataset can be used to improve explanations using a BERT-based classifier . |
Learning to Retrieve Iteratively for In-Context Learning (2024.emnlp-main)
Copied to clipboard
Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, Benjamin Van Durme
| Challenge: | In-context learning is a powerful tool for learning large language models. |
| Approach: | They propose an iterative retrieval framework that empowers retrievers to make iterable decisions through policy optimization. |
| Outcome: | The proposed framework outperforms existing methods on semantic parsing datasets with 4M additional parameters for state encoding. |
PINEAPPLE: Personifying INanimate Entities by Acquiring Parallel Personification Data for Learning Enhanced Generation (2022.coling-1)
Copied to clipboard
Sedrick Scott Keh, Kevin Lu, Varun Gangal, Steven Y. Feng, Harsh Jhamtani, Malihe Alikhani, Eduard Hovy
| Challenge: | Personifications are figures of speech that endow inanimate entities with properties and actions typically seen as requiring animacy. |
| Approach: | They propose to use personification data to train a parallel corpus of personifications . they propose to combine personification-related literalizations with automatic ones . |
| Outcome: | The proposed personification system can generate diverse and creative personifications . it can generate personification-related qualities such as interestingness and animacy . |
Ontologically Faithful Generation of Non-Player Character Dialogues (2024.emnlp-main)
Copied to clipboard
| Challenge: | a key challenge in creating NPC dialogues is that they should serve coherent narratives. |
| Approach: | They propose to use supervised and in-context learning techniques to generate trees of dialogue between video game characters that accurately reflect quest and entity specifications. |
| Outcome: | The proposed model performs well but room for improvement. |
Truth-Conditional Captions for Time Series Data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models with attention mechanisms can generate fluent descriptions of salient patterns in time series, but they often generate factually incorrect descriptions. |
| Approach: | They propose a model which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on *only* the chosen valid program to generate the output text description. |
| Outcome: | The proposed model extracts high-level patterns from the data and generates high precision captions even though it is built on a small space of modules. |
Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing persona-grounded dialog models fail to capture simple implications of given persona descriptions. |
| Approach: | They propose to expand available persona sentences using existing commonsense knowledge bases and paraphrasing resources to imbue dialog models with access to expanded and richer set of persona descriptions. |
| Outcome: | The proposed model outperforms baselines on the Persona-Chat dataset in terms of dialog quality and diversity while achieving persona-consistent and controllable dialog generation. |
Achieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection (2022.acl-long)
Copied to clipboard
| Challenge: | Existing neural dialog models lack specificity and informativeness due to limited knowledge available during training. |
| Approach: | They propose a method to extract relevant knowledge from external sources at decoding time and incorporate it into a dialog response. |
| Outcome: | The proposed method in goal-oriented and knowledge-grounded dialog settings shows that human annotators judge the outputs more engaging and informative compared to responses from prior dialog systems. |
Investigating Robustness of Dialog Models to Popular Figurative Language Constructs (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing dialog models are unable to handle popular figurative language constructs like metaphor and simile when faced with dialog contexts containing figurativ language. |
| Approach: | They propose lightweight solutions to help existing dialog models become more robust to figurative language by simply using an external resource to translate figurativ language to literal (non-figurative) forms while preserving the meaning to the best extent possible. |
| Outcome: | The proposed models show large drops in performance when faced with dialog contexts consisting of figurative language compared to contexts without figurativ language . |
Learning to Describe Differences Between Pairs of Similar Images (D18-1)
Copied to clipboard
| Challenge: | Using crowd-sourced annotations, we generate text descriptions of differences between two images . we use a dataset to generate concise and fluent descriptions of visual data . |
| Approach: | They propose a task of automatically generating text to describe the differences between two images . they crowd-sourced the difference descriptions for pairs of images extracted from video-surveillance footage . |
| Outcome: | The proposed model outperforms models that use attention alone for single-sentence generation and multi-sentent generation. |