Papers by Daniel Khashabi
ICL CIPHERS: Quantifying ”Learning” in In-Context Learning via Substitution Ciphers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that In-Context Learning operates in dual modes . however, disentangling these modes remains a challenging goal . |
| Approach: | They propose a class of task reformulations based on substitution ciphers borrowed from classic cryptography. |
| Outcome: | The proposed model can solve tasks with a BIJECTIVE mapping, but it requires 'deciphering' the latent cipher. |
Evaluating the Evaluators: Are readability metrics good measures of readability? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Plain language summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. |
| Approach: | They conduct a thorough survey of literature on plain language summarization (PLS) and find that traditional readability metrics are not compared to human judgments. |
| Outcome: | The proposed language models better capture deeper measures of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. |
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. |
| Approach: | They propose a retrieval-augmentation method that improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
| Outcome: | The proposed method improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing questions that explicitly describe the process for deriving the answer are often implicit. |
| Approach: | They propose a question answering benchmark where the required reasoning steps are implicit in the question and should be inferred using a strategy. |
| Outcome: | The proposed model is short, topic-diverse, and covers a wide range of strategies. |
arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Literature review tables are essential for summarizing and comparing collections of scientific papers. |
| Approach: | They propose to generate a database of literature review tables from a pool of papers and to model retrieval noise via semantically related but out-of-scope distractor papers verified by human annotators. |
| Outcome: | The proposed method improves over strong baselines while the absolute scores remain modest, underscoring the task’s difficulty. |
Representation Projection Invariance Mitigates Representation Collapse (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning contextualized representations by pre-trained models can lead to representation degradation, which can result in instability, sub-optimal performance, and weak generalization. |
| Approach: | They propose a regularization method to maintain the information content of representations and reduce representation collapse during fine-tuning. |
| Outcome: | The proposed method outperforms baselines on most tasks and improves out-of-distribution performance. |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Augmented Language Models (LLMs) over-rely on task-specific demonstrations that limits their generalizability and computational cost. |
| Approach: | They propose a query-tool grounding algorithm that is generalizable to various tasks . they delegate tool grounding and execution to small language models and LLMs . |
| Outcome: | The proposed algorithm outperforms baselines on 14 datasets and shows it can be generalized to different tasks. |
“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding (D19-1)
Copied to clipboard
| Challenge: | a new study examines temporal commonsense and compares it to human performance on a dataset . a previous study focused on duration, frequency, stationarity and ordering, but not all aspects of temporal similarity have been studied. |
| Approach: | They define five classes of temporal commonsense and use crowdsourcing to develop a new dataset that serves as a test set. |
| Outcome: | The proposed dataset shows that the best current methods are far behind human performance by 20%. |
Core: Robust Factual Precision with Informative Sub-Claim Identification (2025.findings-acl)
Copied to clipboard
Zhengping Jiang, Jingyu Zhang, Nathaniel Weir, Seth Ebner, Miriam Wanner, Kate Sanders, Daniel Khashabi, Anqi Liu, Benjamin Van Durme
| Challenge: | Using the Decompose-Then-Verify framework, such as FActScore, can be manipulated by adding obvious or repetitive subclaims to artificially inflate scores. |
| Approach: | They propose a decomposition-based tool called Core to filter subclaims based on their uniqueness and informativeness. |
| Outcome: | The proposed evaluation framework supports easy and modular use of Core and various decomposition strategies. |
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies (2024.emnlp-main)
Copied to clipboard
Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Murari Tiyyala, Nicholas Andrews, Daniel Khashabi
| Challenge: | Analogical reasoning is an important part of human communication, says a new study . a benchmark to determine analogical reasoning ability in language models is needed . |
| Approach: | They propose to benchmark analogical reasoning ability in language models by collecting 340 analogies from human writings. |
| Outcome: | The proposed benchmark aims to determine analogical reasoning ability in language models. |
Temporal Common Sense Acquisition with Minimal Supervision (2020.acl-main)
Copied to clipboard
| Challenge: | Temporal common sense is crucial for understanding natural language, but its acquisition is challenging . human annotation on such concepts is costly and often not made explicit in text . |
| Approach: | They propose a method that exploits explicit and implicit mentions of temporal common sense to build a temporal similarity language model. |
| Outcome: | The proposed model gives better predictions of various dimensions of temporal common sense than the standard BERT. |
Jailbreak Distillation: Renewable Safety Benchmarking (2025.findings-emnlp)
Copied to clipboard
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang, A S M Iftekhar, Ahmed Magooda, Benjamin Van Durme, Daniel Khashabi, Kyle Jackson
| Challenge: | a new framework for safety benchmarking is being developed for large language models . the framework "distills" jailbreak attacks into high-quality and easily-updatable safety benchmarks . |
| Approach: | They propose a framework that "distills" jailbreak attacks into high-quality safety benchmarks. |
| Outcome: | The proposed framework "distills" jailbreak attacks into high-quality safety benchmarks . it requires minimal human effort to rerun the pipeline and produce updated benchmarks. |
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2022.naacl-main)
Copied to clipboard
| Challenge: | a pretrained model is optionally adapted through domain-specific pretraining, followed by task-specific finetuning. |
| Approach: | They establish a suite of eight tasks across different domains to quantify the effects of temporal misalignment in modern NLP systems. |
| Outcome: | The proposed tasks are based on eight domains and periods of time spanning five years or more and show that they have stronger effects than previous studies. |
ParsiNLU: A Suite of Language Understanding Challenges for Persian (2021.tacl-1)
Copied to clipboard
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, Yadollah Yaghoobzadeh
| Challenge: | Despite progress in natural language understanding, most progress is concentrated on resource-rich languages like English . despite high-quality benchmarks, there are few available NLU datasets for Persian language . |
| Approach: | They propose a benchmark for Persian language that includes a range of language understanding tasks . they present their results on monolingual and multilingual pre-trained language models . |
| Outcome: | The proposed benchmarks compare human performance with monolingual and multilingual models on Persian language with high quality evaluation datasets. |
RORA: Robust Free-Text Rationale Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing metrics rely on degree to which rationale supports a label, but they fail to evaluate rationales that inadvertently leak the label. |
| Approach: | They propose a RObust free-text RAtionale evaluation against label leakage that quantifies the new information supplied by a rationale to justify the label. |
| Outcome: | The proposed evaluation outperforms existing methods in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. |
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023.acl-long)
Copied to clipboard
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
| Challenge: | Large “instruction-tuned” language models depend heavily on human-written instruction data . this limited quantity, diversity, and creativity hinders the generality of the tuned model . |
| Approach: | They propose a framework for improving instruction-following capabilities of pretrained language models by bootstrapping off their own generations. |
| Outcome: | The proposed framework outperforms existing public instruction datasets by 5% . it generates instructions, input, and output samples, then filters invalid or similar ones . |
UNQOVERing Stereotyping Biases via Underspecified Questions (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language embeddings have been shown to have stereotyping biases, but how these biase affecting downstream question answering models remains unexplored. |
| Approach: | They propose a general framework to probe biases through underspecified questions by building minimal context and building minimal questions. |
| Outcome: | The proposed framework isolates two types of reasoning errors and identifies stereotyping biases in gender, nationality, ethnicity, and religion classes. |
Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. |
| Approach: | They propose to examine LLMs' long-context generalizations by probing their hidden representations. |
| Outcome: | The proposed models excel at processing extended contexts while preserving their positional bias. |
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)
Copied to clipboard
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi
| Challenge: | Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks. |
| Approach: | They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve. |
| Outcome: | The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy. |
Query Decomposition for RAG: Balancing Exploration-Exploitation (2026.eacl-long)
Copied to clipboard
Roxana Petcu, Kenton Murray, Daniel Khashabi, Evangelos Kanoulas, Maarten de Rijke, Dawn Lawrie, Kevin Duh
| Challenge: | Complex user queries often involve the exclusion of information, negation, or missing entities. |
| Approach: | They propose to decompose user requests into subqueries, retrieve potentially relevant documents for each and then aggregate them to generate an answer. |
| Outcome: | The proposed method achieves 35% gain in document-level precision and 15% increase in -nDCG . it also improves the downstream task of long-form generation. |
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (N18-1)
Copied to clipboard
| Challenge: | Using a dataset of 6,500+ questions, we found that human solvers achieved an F1-score of 88.1%. |
| Approach: | They propose a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. |
| Outcome: | The proposed reading comprehension challenge is based on a reading comprehension dataset with 6,500+ questions and 1000+ paragraphs across 7 domains. |
PerspectroScope: A Window to the World of Diverse Perspectives (P19-3)
Copied to clipboard
| Challenge: | PerspectroScope is a web-based system that lets users query a discussion-worthy natural language claim . |
| Approach: | They propose a web-based system which lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
| Outcome: | The proposed system lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
Benchmarking Language Model Creativity: A Case Study on Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies on LLM creativity evaluation focus on open-ended generation tasks . however, the degree to which LLMs possess and utilize creativity for problem-solving remains unclear . |
| Approach: | They propose a framework for quantifying LLM creativity that incorporates design ingredients . they introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions . |
| Outcome: | The proposed framework quantifies creativity in LLMs on Codeforces problems . it also finds that even the most creative model fails to demonstrate human-like creativity . |
Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems (2026.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to improve long-context understanding of large language models lack scalability and stability. |
| Approach: | They challenge the assumption that SC’s benefits generalize to long-context settings . they find that persistent position bias degrades performance on long-consistency tasks . |
| Outcome: | The proposed approach fails to improve and actively degrades performance on long-context tasks. |
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) may hallucinate and generate false information despite pre-training on factual data. |
| Approach: | They propose a new evaluation metric that measures the extent to which model-produced answers are directly found in underlying text corpora. |
| Outcome: | The proposed evaluation metric measures the extent to which model-produced answers are directly found in underlying text corpora. |
k-SemStamp: A Clustering-Based Semantic Watermark for Detection of Machine-Generated Text (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent watermarked generation algorithms inject detectable signatures during language generation to facilitate post-hoc detection. |
| Approach: | They propose a watermark which assigns signatures to each watermarked sentence according to locality-sensitive hashing (LSH) they propose k-SemStamp, which uses kmeans clustering to partition the semantic space with awareness of inherent semantic structure. |
| Outcome: | The proposed watermark improves its robustness and sampling efficiency while preserving the generation quality, making it more effective for machine-generated text detection. |
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022.acl-long)
Copied to clipboard
| Challenge: | Despite the success of supervised learning, models often struggle with generalization across tasks. |
| Approach: | They propose to use crowdsourcing instructions to build a model that learns a new task by understanding the human-readable instructions that define it. |
| Outcome: | The proposed model can learn from seen tasks and generalize to unseen tasks given its natural crowdsourcing instructions. |
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, Maarten Sap
| Challenge: | Existing dialogue systems fail to respond properly to potentially unsafe user utterances . existing systems either ignore or passively agree with unsafe content . |
| Approach: | They introduce a dataset to teach conversational agents to respond to problematic content following social norms. |
| Outcome: | The proposed dataset shows that ProsocialDialog generates more socially acceptable dialogues than existing models. |
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to verify text accuracy provide no guarantees on their correctness . a new method to improve LLMs' verifiability is to use quotes to ground models . |
| Approach: | They propose a method that allows models to quote verbatim statements from trusted sources . they leverage a fast membership inference function to verify text against trusted corpora . |
| Outcome: | The proposed method significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
Reframing Instructional Prompts to GPTk’s Language (2022.findings-acl)
Copied to clipboard
| Challenge: | Using reframing techniques, we find that instructional prompts are easier to follow for Language Models (LMs) |
| Approach: | They propose reframing techniques for manual reformulation of prompts into more effective ones . they compare performance of LMs prompted with reframed instructions on 12 NLP tasks . |
| Outcome: | The reframing techniques used for prompt reformulation improve performance on 12 tasks . the techniques boost performance on LMs with different sizes compared with original prompts . |
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2023.acl-short)
Copied to clipboard
| Challenge: | omnipresence of large pre-trained language models has fueled concerns regarding systematic biases carried over from underlying data into the applications they are used in. |
| Approach: | They propose to compare social biases with non-social biase masked by alternate constructions that maintain the essence of their social bias. |
| Outcome: | The proposed benchmarks underestimate or overestimate the social bias in a given model. |
CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers? (2025.findings-emnlp)
Copied to clipboard
Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, Benjamin Van Durme
| Challenge: | CLAIMCHECK is an annotated dataset of NeurIPS 2023 and 2024 submissions and reviews from OpenReview. |
| Approach: | They annotate NeurIPS 2023 and 2024 submissions and reviews for weaknesses and dispute them for fine-grained labels of validity, objectivity, and type of the identified weaknesses. |
| Outcome: | The proposed dataset is richly annotated by ML experts for weaknesses statements in the reviews and the claims that they dispute, as well as fine-grained labels of validity, objectivity, and type of the identified weaknesses. |
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation (2022.emnlp-main)
Copied to clipboard
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel Weld
| Challenge: | Effective human evaluation of text generation tasks remains an important, open area for research. |
| Approach: | They propose a system for running standardized human evaluations across different generation tasks. |
| Outcome: | The proposed system produces standardized human evaluations across tasks . it crowdsources predictions and ranks systems on leaderboards . the proposed system is not reproducible over time and different annotator populations . |
Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses (2020.acl-main)
Copied to clipboard
| Challenge: | Empirical research in natural language processing has adopted a narrow set of principles for assessing hypotheses . alternative approaches to assess hypothese rely on p-value computation, which suffers from several known issues. |
| Approach: | They propose to compare different methods for assessing hypotheses . they argue that practitioners should first decide their target hypothesis before choosing a method . |
| Outcome: | The proposed method differs from other methods, but is not widely used in NLP . the proposed method is based on a p-value computation, but has a small gap in accuracy . |
Hey AI, Can You Solve Complex Tasks by Talking to Agents? (2022.findings-acl)
Copied to clipboard
| Challenge: | CommaQA is a benchmark for learning to solve complex tasks by communicating with existing agents in natural language. |
| Approach: | They propose a synthetic benchmark with three complex reasoning tasks designed to be solved by communicating with existing QA agents. |
| Outcome: | The proposed model outperforms models that learn to communicate with agents without auxiliary supervision or data. |
Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims (N19-1)
Copied to clipboard
| Challenge: | a number of fact checking techniques are used to identify and eliminate biases in text data. |
| Approach: | They propose to use search engines to expand and diversify a dataset of claims, perspectives and evidence to address a selection bias. |
| Outcome: | The proposed approach outperforms existing methods in a language understanding task. |
More Bang for Your Buck: Natural Perturbation for Robust Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Creating large datasets to train NLP models is becoming increasingly expensive. |
| Approach: | They propose to use a question-answering dataset to expand a training set using human-driven perturbations instead of rule-based machine perturbations. |
| Outcome: | The proposed approach improves on a question-answering dataset with human-driven perturbations. |
Certified Mitigation of Worst-Case LLM Copyright Infringement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are trained on vast datasets that include copyrighted material or content with usage restrictions. |
| Approach: | They propose a "copyright takedown" method that provides certified copyright take down . they use a combination of inference-time and rewriting techniques to transform potentially infringing segments . |
| Outcome: | The proposed method reduces infringement risk, preserves utility, and accommodates different levels of enforcement stringency with adaptive abstention. |
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Manually "engineering" prompts for large language models can be laborious and time-intensive. |
| Approach: | They propose a new metric to quantify the expected utility of a language prompt. |
| Outcome: | The proposed metric outperforms previous prompt selection metrics with 10% increase in Pearson correlation across 6 classification benchmarks and the prompt selected by the proposed meter gains 5% higher accuracy than previous metrics. |
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021.findings-acl)
Copied to clipboard
| Challenge: | a new task is proposed to modify a model's unethical behavior by communicating context-specific principles of ethics and equity to it. |
| Approach: | They propose a task to amend a questionanswering model's unethical behavior . they use social stereotypes to augment social stereotype models with ethical interventions . |
| Outcome: | The proposed task improves model behavior but fails to generalize to new domains . it is based on a zero-shot evaluation of social stereotypes in reading comprehension systems . |
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets (2025.naacl-long)
Copied to clipboard
| Challenge: | a lack of data across domains creates significant imbalances in training data sizes . a recent study shows that temperature sampling and scaling are equivalent but differ under stochastic gradient descent due to differences in gradient variance. |
| Approach: | They propose a method that upsamples low-resource languages and upweights their loss functions to address this disparity. |
| Outcome: | The proposed method competes effectively with existing data re-weighting techniques while offering computational efficiency. |
GooAQ: Open Question Answering with Diverse Answer Types (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on niche answer types, mainly short responses and, in a few cases, long responses, has failed to adequately address the answer diversity of questions. |
| Approach: | They propose to use Google's autocomplete feature to collect questions from a large-scale dataset with a variety of answer types to facilitate further research on improving QA with diverse response types. |
| Outcome: | The proposed model produces naturalistic questions that are short and expressed using simple language. |
Challenging the Evaluator: LLM Sycophancy Under User Rebuttal (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often exhibit sycophancy, distorting responses to align with user beliefs. |
| Approach: | They investigate why LLMs exhibit sycophancy when challenged in subsequent conversational turns, yet perform well when evaluating conflicting arguments presented simultaneously? |
| Outcome: | The proposed models are more likely to endorse a user’s counterargument when framed as a follow-up from a users, rather than when both responses are presented simultaneously for evaluation. |
CogCompNLP: Your Swiss Army Knife for NLP (L18-1)
Copied to clipboard
Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, Dan Roth
| Challenge: | a corpus-reader module supports popular corpora, feature extraction and annotation modules for semantic and syntactic tasks. |
| Approach: | They propose a library that provides modules to address different challenges . they provide a corpus-reader module that supports popular corpora in the NLP community . |
| Outcome: | The proposed library simplifies the process of design and development of NLP applications by providing modules to address different challenges. |
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts (2024.findings-acl)
Copied to clipboard
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, Daniel Khashabi
| Challenge: | Recent studies show that malicious prompt instructions could solicit objectionable content from LLMs. |
| Approach: | They compare how state-of-the-art LLMs respond to malicious prompts in different languages . they find that LLM's generate unsafe responses more often when a prompt is written in a lower-resource language . |
| Outcome: | The proposed model can generate unsafe responses more often when a malicious prompt is written in a lower-resource language, and less irrelevant responses when written in lower-source languages. |
SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation (2024.naacl-long)
Copied to clipboard
Abe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, Yulia Tsvetkov
| Challenge: | Existing watermarked generation algorithms employ token-level designs and are vulnerable to paraphrase attacks. |
| Approach: | They propose a sentence-level watermarking algorithm that uses locality-sensitive hashing to partition the semantic space of sentences. |
| Outcome: | The proposed algorithm is more robust than the existing state-of-the-art method on paraphrasers and domains, while posing only minor degradations to SemStamp. |
Zero-Shot Open Entity Typing as Type-Compatible Grounding (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to entity typing have limited flexibility to transfer across text genres and generalize to new type taxonomies. |
| Approach: | They propose a zero-shot entity typing approach that requires no annotated data and can flexibly identify newly defined types. |
| Outcome: | The proposed system outperforms state-of-the-art supervised NER systems on a broad range of datasets and on 'biological domain' it is competitive with supervised systems and outperformed on out-of training datasets. |
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2022.naacl-main)
Copied to clipboard
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, Yejin Choi
| Challenge: | Existing paradigms for text generation are left-to-right decoding from autoregressive language models. |
| Approach: | They propose a decoding algorithm that incorporates heuristic estimates of future cost that are efficient for large-scale language models. |
| Outcome: | The proposed method outperforms baselines on five generation tasks and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. |
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to decompose complex tasks into simpler ones do not require annotated decompositions. |
| Approach: | They propose a framework for building interpretable systems that learn to solve complex tasks by decomposing existing models into simpler ones solvable by existing models. |
| Outcome: | The proposed framework is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work. |