| Challenge: | Large Language Models (LLMs) have become increasingly prevalent in the field of Natural Language Processing (NLP), achieving unprecedented performance across linguistic tasks. |
| Approach: | They propose a framework to quantify and analyze context-driven over-refusal . they find that over-fusals depend on the task, system prompts, model family, and the number of retrieved documents. |
| Outcome: | The proposed framework quantifyes and analyzes the concept of context-driven over-refusal on two public corpora. |
Similar Papers
SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steering (2026.acl-long)
Copied to clipboard
| Challenge: | Current safety alignment methods fail to identify intended benign task before refusing to respond. |
| Approach: | They propose a method that uses inference-time trajectory-shifting to guide model behavior . they show that LLMs persist in refusing inputs containing harmful content . |
| Outcome: | The proposed approach reduces over-refusals with minimal impact on utility. |
Dynamic Evaluation for Oversensitivity in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks rely on static datasets that degrade over time as models evolve, leading to data contamination and diminished evaluative power. |
| Approach: | They construct a framework that generates model-specific challenging datasets and aggregates them across diverse LLM families. |
| Outcome: | The framework captures emerging defensive patterns and aligns with each model’s unique behavior. |
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence. |
| Approach: | They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function . |
| Outcome: | The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes. |
PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are useful for low-resource scenarios and time-restricted applications. |
| Approach: | They propose a large-scale evaluation tool for large language models that uses prompts . they evaluate 720 prompt templates for open-source LLM-based metrics on MT and summarization datasets a 6.6M evaluations. |
| Outcome: | The proposed model evaluates 720 prompt templates on machine translation and summarization datasets. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Born Pragmatic, Trained to Hallucinate? Quantifying the Origins of Contextual Bias in LLMs via the PaCE Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel at capturing communicative intent, but they have a side effect: pragmatic hallucination. |
| Approach: | They propose a benchmark to quantify the impact of pragmatic hallucination on large language models . they propose RLHF and SFT to induce a strong tendency for pragmatic over-attribution . |
| Outcome: | The proposed model outperforms existing models in predicting pragmatic hallucinations . the evaluations show that current alignment paradigms lack precise control over pragmatic boundaries . |
Investigating Context Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style (2025.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation improves Large Language Models (LLMs) by integrating external information into the response generation process. |
| Approach: | They investigate the impact of memory strength and evidence presentation on LLMs’ receptiveness to external evidence by measuring the divergence in LLM responses to different paraphrases of the same question. |
| Outcome: | The proposed method improves Large Language Models (LLMs) by integrating external information into the response generation process. |
Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for mitigating over-refusal can't maintain low refusal ratio for harmless queries while keeping high for malicious queries. |
| Approach: | They propose a model-agnostic approach to mitigate over-refusal in large language models . they propose an adaptive contrastive decoding strategy that incorporates or removes the refusal token distribution . |
| Outcome: | The proposed approach reduces the refusal ratio for over-refusal queries by 10.35% while increasing the refusal rate for malicious queries by 0.13%. |
Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often refuse to answer legitimate queries, causing models to treat many reasonable prompts as potentially risky. |
| Approach: | They propose a framework that automatically generates and selects overrefusal prompts near the safety boundary. |
| Outcome: | The proposed framework identifies and curates boundary-aligned prompts, enabling more effective and targeted mitigation of overrefusal. |
Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to jailbreak attacks that exploit weaknesses in traditional safety alignment. |
| Approach: | They propose a framework that trains models to engage in explicit safe reasoning before response . they propose RATIONAL, which allows models to reject harmful prompts while providing meaningful and context-aware responses. |
| Outcome: | The proposed framework fine-tunes models to reason about query intent, ethics, and potential harm. |