Papers by Renyi Qu
Is Semantic Chunking Worth the Computational Cost? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Retrieval-Augmented Generation (RAG) systems have popularized semantic chunking. |
| Approach: | They evaluate the effectiveness of semantic chunking using three common retrieval tasks . they find that the computational costs associated with semantic chunks are not justified by consistent performance gains. |
| Outcome: | The proposed semantic chunking approach is not able to deliver consistent performance gains in three retrieval-related tasks. |
Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (2025.emnlp-industry)
Copied to clipboard
Manveer Singh Tamber, Forrest Sheng Bao, Chenyu Xu, Ge Luo, Suleman Kazi, Minseok Bae, Miaoran Li, Ofer Mendelevitch, Renyi Qu, Jimmy Lin
| Challenge: | Large language models (LLMs) excel in various tasks, but often produce hallucinations . retrieved contexts, misrepresent information, or generate outright contradictions . |
| Approach: | They propose a framework that measures hallucination faithfulness of large language models . they introduce a leaderboard that leverages diverse human-annotated hallucinian examples . |
| Outcome: | The proposed framework improves hallucination evaluations by leveraging human-annotated examples. |
Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains where safety depends on how a system answers . out-of-domain (OOD) queries can impair performance and safety . |
| Approach: | They propose to use lightweight, KB-aligned OOD detection as an always-on gate for RAG systems. |
| Outcome: | The proposed method scores queries in a compact subspace selected either by explained-variance retention (EVR) or by a separability-driven -test ranking. |
FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs (2025.naacl-short)
Copied to clipboard
Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, Amin Ahmad
| Challenge: | Existing evaluations of hallucinations in large language models suffer from a lack of diversity and recency in the LLM and LLM families considered. |
| Approach: | They propose a summarization hallucination benchmark that challenges models to disagree on hallucines . they use models to generate answers or summaries from textual input . |
| Outcome: | The proposed model combines the best of 10 modern LLMs with ground truth annotations. |
Conditioning on Dialog Acts improves Empathy Style Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research has focused on empathetic response generation, but it is not applicable to more sensitive cases such as medicine and therapy where the content of the responses requires the supervision of medical experts. |
| Approach: | They propose two new style transfer strategies that use only examples of the target style and dialog-act-conditioned prompting to make a sentence more empathetic. |
| Outcome: | The proposed methods improve empathy more effectively while maintaining semantic similarity and preserving both semantics and the dialog-act type. |