Papers by Abhilasha Ravichander
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)
Copied to clipboard
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
| Challenge: | In this paper, we examine whether pretrained language models are consistent with factual knowledge. |
| Approach: | They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue . |
| Outcome: | The proposed model improves consistency and shows that it is effective. |
Agent Lumos: Unified and Modular Training for Open-Source Language Agents (2024.acl-long)
Copied to clipboard
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, Bill Yuchen Lin
| Challenge: | Lumos is a framework for training open-source agents on complex interactive tasks. |
| Approach: | They propose a framework for training open-source LLM-based agents called Lumos . Lumos features a learnable, unified and modular architecture with a planning module that learns high-level subgoal generation and a grounding module trained to translate these into the actions using various tools in the execution module. |
| Outcome: | The framework outperforms open-source agents on QA and web tasks. |
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can’t Answer? (2025.naacl-short)
Copied to clipboard
| Challenge: | Question answering (QA) is a popular task, but we test both separately . a recent study found that LLMs are less accurate in numerical RQA than RQA . |
| Approach: | We run 16 LLMs on QA and RQA with trivia questions/answers . they find question and answer types that lead to RQA errors and suggest improvements . |
| Outcome: | The results show that LLMs are less accurate in RQA for numerical answers than RQA . RQA errors correlate with question difficulty and inversely correlate with answer frequencies . |
Exploring Numeracy in Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing word embeddings are inadequate at capturing numerical properties of numbers. |
| Approach: | They propose to use word embeddings to capture numerical properties of numbers . they hope to develop methods which better capture numeric properties . |
| Outcome: | The proposed models lack the ability to capture numeric properties of numbers, the authors show . their findings provide a starting point for the development of better models . |
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives (D19-1)
Copied to clipboard
| Challenge: | Privacy policies are long and complex documents that are difficult for users to read and understand. |
| Approach: | They present a corpus of 1750 questions about privacy policies of mobile applications and over 3500 expert annotations of relevant answers. |
| Outcome: | The proposed corpus of 1750 questions on privacy policies shows that a strong neural baseline underperforms human performance by almost 0.3 F1 on PrivacyQA. |
Erratum: Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)
Copied to clipboard
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
| Challenge: | During production of this paper, an error was introduced to the formula on the bottom of the right column of page 1020. |
| Approach: | the formula was changed in the last two terms of the paper . |
| Outcome: | the correct formula is now available on the web. |
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing studies have found that BERT can correctly retrieve noun hypernyms in cloze tasks, but this does not correspond to systematic knowledge in BERT. |
| Approach: | They propose to use BERT to probe for hypernymy knowledge encoded in representations for cloze tasks to find out whether it is systematic or not . |
| Outcome: | The proposed model can retrieve hypernyms in cloze tasks, but not systematic knowledge in BERT. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
A Tale of Two Regulatory Regimes: Creation and Analysis of a Bilingual Privacy Policy Corpus (2022.lrec-1)
Copied to clipboard
Siddhant Arora, Henry Hosseini, Christine Utz, Vinayshekhar Bannihatti Kumar, Tristan Dhellemmes, Abhilasha Ravichander, Peter Story, Jasmine Mangat, Rex Chen, Martin Degeling, Thomas Norton, Thomas Hupperich, Shomir Wilson, Norman Sadeh
| Challenge: | With the introduction of new privacy regulations, disclosures made by the same organization are not always the same in different languages. |
| Approach: | They propose a language annotation scheme to capture nuances of two new privacy regulations, namely the EU’s GDPR and California’s CCPA/CPRA. |
| Outcome: | The proposed method captures the nuances of two new privacy regulations and compares them to a corpus of 64 privacy policies in English and 91 in German with manual annotations for 8K and 19K fine-grained data practices. |
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Negation is fundamental to human communication. |
| Approach: | They propose a dataset which requires reasoning about implications of negated statements in paragraphs . they collect paragraphs with diverse negation cues and crowdworkers ask questions about implications . |
| Outcome: | The first dataset in english requires reasoning about implications of negated statements in paragraphs . it features 14,182 question-answer pairs with over 200 unique negation cues based on crowd-workers . the best performing model achieves only 42% on consistency metric, well below human performance of 81%. |
Breaking Down Walls of Text: How Can NLP Benefit Consumer Privacy? (2021.acl-long)
Copied to clipboard
| Challenge: | Privacy policies are long and complex documents that are difficult for users to read and comprehend. |
| Approach: | They propose language technologies to help users reclaim control over their privacy . they highlight many remaining opportunities to develop more precise or nuanced language technologies . |
| Outcome: | The proposed language technologies can address the privacy information gap . they can be more precise or nuanced in the way they use the text of privacy policies. |
MacGyver: Are Large Language Models Creative Problem Solvers? (2024.naacl-long)
Copied to clipboard
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, Faeze Brahman
| Challenge: | a new study examines the creative problem-solving capabilities of modern LLMs . it provides insight into the constrained problem- solving capabilities of both humans and AI . |
| Approach: | They use an automatically generated dataset to compare and contrast LLMs and humans to find out their creative problem-solving abilities. |
| Outcome: | The proposed dataset compares LLMs and humans in a constrained setting . it shows that humans excel in tasks they are familiar with but struggle with domain-specific knowledge . |
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (2025.acl-long)
Copied to clipboard
| Challenge: | generative large language models produce hallucinations that are not aligned with world knowledge or input context. |
| Approach: | They propose a hallucination benchmark framework that measures hallucinism in large language models . they evaluate 150,000 generations from 14 language models and find they are riddled with hallucinos . |
| Outcome: | The proposed framework evaluates 150,000 generations from 14 language models. |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
Stress Test Evaluation for Natural Language Inference (C18-1)
Copied to clipboard
| Challenge: | Existing models perform well at standard datasets for NLI, achieving impressive results across different genres of text. |
| Approach: | They propose to use automatic stress tests to evaluate models' ability to make inferential decisions. |
| Outcome: | The proposed model performs well across genres of text, but lacks the ability to make inferential decisions. |
What Has Been Lost with Synthetic Evaluation? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study evaluated the validity and difficulty of large language models for evaluation benchmarks . large language model evaluation benchmarking is challenging and requires specific phenomena to be addressed . |
| Approach: | They compare LLM-generated reasoning-over-text benchmarks to those generated through crowdsourcing . they find they are *less challenging for LLMs* than their human-authored counterparts . |
| Outcome: | The results show that LLMs can generate variants that are valid according to annotation guidelines, but less challenging than human-authored counterparts. |
Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (2025.naacl-long)
Copied to clipboard
Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Maria Antoniak, Bill Yuchen Lin, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin Choi
| Challenge: | Lack of transparency in training data is limiting external oversight and inspection of LLMs for issues such as copyright infringement and data contamination. |
| Approach: | They propose a method to identify training data known to proprietary LLMs without requiring access to model weights or token probabilities by using information-guided probes. |
| Outcome: | The proposed method can identify training data known to proprietary LLMs without access to model weights or token probabilities. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts . |
| Approach: | They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone . |
| Outcome: | The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain. |