Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG (2025.acl-long)
Copied to clipboard
Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Zilei Wang, Weiqiang Wang, Liang Wang
| Challenge: | Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources. |
| Approach: | They propose a method that conditions large language models to generate answers even in the absence of reliable knowledge. |
| Outcome: | The proposed approach balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems. |
Similar Papers
Uplift-RAG: Uplift-Driven Knowledge Preference Alignment for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to estimate document utility rely on downstream generation performance, which conflates the influence of external documents with the intrinsic knowledge of the LLM. |
| Approach: | They propose an uplift-based definition of document utility that quantifies each document’s marginal benefit over the LLM’s internal knowledge. |
| Outcome: | The proposed framework improves the performance of the LLM by incorporating external retrieved documents into the model. |
KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models (2025.findings-emnlp)
Copied to clipboard
Zhen Zhang, Xinyu Wang, Yong Jiang, Zile Qiao, Zhuo Chen, Guangyu Li, Feiteng Mu, Mengting Hu, Pengjun Xie, Fei Huang
| Challenge: | Retrieval-augmented generation (RAG) is employed to tackle these challenges . a Knowledge Boundary Model (KBM) is used to express the known/unknown of a given question . |
| Approach: | They propose a Knowledge Boundary Model to express the known/unknown of a given question . they find that not all questions need to trigger RAG to improve performance . |
| Outcome: | The proposed model reduces time and computational costs by retrieving parts of unknown knowledge . the proposed model can express the known/unknown of a given question and determine whether a RAG needs to be triggered . |
GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to retrieve information from large language models (LLMs) but they fail to address the preference gap between retrievers and LLMs. |
| Approach: | They propose a retrieval module that dynamically injects retrieved information into the input context of large language models (LLMs) This approach aligns the retriever’s and LLM’s preferences by defining a new metric, “gain”, which measure how well an input passage contributes to correct outputs. |
| Outcome: | The proposed approach has shown significant success in various NLP tasks, but there is a preference gap between retrievers and LLMs. |
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models may not follow human instructions and produce toxic, hallucinated, or biased content. |
| Approach: | They propose a disperse-then-merge framework that dispersers instruction-following data into portions and trains multiple sub-models using different data portions. |
| Outcome: | The proposed framework outperforms data curation and training regularization on standard knowledge and reasoning benchmarks. |
RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle to evaluate the correctness of non-parametric knowledge when it differs from internal memorization, leading to knowledge conflicts during response generation. |
| Approach: | They propose a lightweight alignment method to leverage multi-source knowledge based on retrieval relevance. |
| Outcome: | Experiments on four datasets show that the proposed method outperforms RAG by 4-10% in accuracy without any extra component. |
Fair RAG: End-to-End Fairness Across Retrieval and Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can amplify demographic bias by generating skewed context . prior work treats fairness in retrieval or generation in isolation, leaving end-to-end fairness underexplored . |
| Approach: | They propose a pipeline that jointly controls both retrieval and generation stages . large language models can handle a broad set of inference tasks, they argue . |
| Outcome: | The proposed pipeline reduces retriever-side skew and achieves lowest generator-side disparity while preserving utility. |
No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-augmented generation is widely adopted for its effectiveness and cost-efficiency in mitigating hallucinations. |
| Approach: | They propose a practical three-level threat model from the perspective of user fairness awareness. |
| Outcome: | The proposed model shows that RAG can undermine fairness alignment without fine-tuning or retraining. |
PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize RAG generators fail to align with RAG requirements thoroughly. |
| Approach: | They propose a method for optimizing the RAG generator from multiple preference perspectives to align with RAG requirements comprehensively. |
| Outcome: | The proposed method improves the performance of RAG generators by incorporating retrieved documents into the prompt. |
Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have not linked the behavior of retrieval augmented generation (RAG) with imperfect retrieval, including irrelevant, misleading, or even malicious information. |
| Approach: | They propose an approach that integrates external knowledge with source-awareness to overcome imperfect retrieval errors in RAG. |
| Outcome: | The proposed approach is superior to previous robustness-enhanced approaches under the worst-case scenario. |
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training (2026.findings-acl)
Copied to clipboard
Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes, Yang Zhang
| Challenge: | Misaligned large language models can magnify harm by exploiting them to undermine safety . et al., 2022b; Bai e.t., 2023): misalignment, realignment and model-specific resistance are important . |
| Approach: | They evaluate four methods to identify a mechanism asymmetry between attack and defense . they find that ORPO is most effective for misalignment, but DPO excels in realignment . |
| Outcome: | The proposed methods show a mechanism asymmetry between attack and defense . the proposed methods excel in realignment, but at the expense of model utility . |