Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as rerankers, but their ranking behavior can be steered by small, natural-sounding prompts. |
| Approach: | They propose a token optimization method that crafts concise textual perturbations to consistently promote a target item in LLM-generated rankings. |
| Outcome: | The proposed method outperforms state-of-the-art base-lines and is hard to detect. |
Similar Papers
ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have demonstrated remarkable performance in document reranking tasks. |
| Approach: | They propose a two-stage training approach for document reranking using reinforcement learning and fine-grained score learning. |
| Outcome: | The proposed approach outperforms open-source and proprietary reranking models on BEIR benchmark. |
DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | DeAR is an open-source framework that decouples the tasks of LLMs with holistic cross-document analysis. |
| Approach: | They propose an open-source framework that decouples relevance scoring with holistic cross-document analysis. |
| Outcome: | The proposed framework outperforms open-source frameworks in QA and open-domain QA. |
FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing listwise LLMs lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. |
| Approach: | They propose a listwise LLM reranking approach that leverages the first generated identifier to obtain a ranked ordering of the candidates. |
| Outcome: | The proposed approach accelerates inference by 50% while maintaining robust ranking performance with gains across BEIR benchmark. |
REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs face ranking uncertainty, unstable top-k recovery, and high token cost due to token-intensive prompting. |
| Approach: | They propose a re-ranking framework that captures uncertainty and refines LLM-derived relevance through recursive Bayesian updates. |
| Outcome: | The proposed framework outperforms state-of-the-art re-rankers while reducing token usage and latency. |
PRP-Graph: Pairwise Ranking Prompting to LLMs with Graph Aggregation for Effective Text Re-ranking (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for pairing ranking prompting only output the same label for comparison results of different confidence intervals without considering the uncertainty of pairwise comparison. |
| Approach: | They propose a pairwise ranking prompting approach that exploits the output probabilities of target labels to capture the degree of certainty of comparison results. |
| Outcome: | The proposed method shows strong robustness and acceptable efficiency on the BEIR benchmark. |
EcoRank: Budget-Constrained Text Re-ranking Using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated dominant performance in text re-ranking. |
| Approach: | They propose a suite of budget-constrained methods to perform text re-ranking using LLMs. |
| Outcome: | The proposed method outperforms other budget-aware methods on four datasets. |
GroupRank: A Groupwise Paradigm for Effective and Efficient Passage Reranking with LLMs (2026.findings-acl)
Copied to clipboard
Meixiu Long, Duolin Sun, Dan Yang, Yihan Jiao, Lei Liu, Jiahai Wang, Binbin Hu, Yue Shen, Jie Feng, Zhehao Tan, Junjie Wang, Lianzhen Zhong, Jian Wang, Peng Wei, Jinjie Gu
| Challenge: | Existing rankers excel in lexical-matching scenarios, while they struggle with complex queries requiring deep reasoning. |
| Approach: | They propose a new paradigm that balances flexibility and context awareness to unlock the full potential of groupwise reranking. |
| Outcome: | The proposed approach achieves a state-of-the-art 65.2 NDCG@10 on BRIGHT and surpasses baselines by 2.1 points on R2MED while delivering a 6.4 inference speedup. |
Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing pointwise LLMs provide noisy or biased answers for documents that are partially relevant to the query. |
| Approach: | They propose to incorporate fine-grained relevance labels into the LLM prompt . they propose to better differentiate between documents with different levels of relevance . |
| Outcome: | The proposed model can differentiate between documents with different levels of relevance to the query and derive a more accurate ranking. |
Explain then Rank: Scale Calibration of Neural Rankers Using Natural Language Explanations from LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Neural ranking models produce the final document scores, but they are often treated as transient information and only the relative orderings are preserved to produce a ranking. |
| Approach: | They propose to exploit large language models (LLMs) to provide relevance and uncertainty signals for these neural text rankers to produce scale-calibrated scores through Monte Carlo sampling of natural language explanations (NLEs). |
| Outcome: | The proposed approach outperforms previous calibration methods and LLM-based methods for ranking, calibration, and query performance prediction tasks. |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024.findings-naacl)
Copied to clipboard
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky
| Challenge: | Existing methods to rank documents using large language models do not understand these challenging ranking formulations. |
| Approach: | They propose to use Pairwise Ranking Prompting to improve ranking performance . they propose to outperform fine-tuned baseline rankers on benchmark datasets . |
| Outcome: | The proposed technique outperforms supervised baselines on benchmark datasets and outperformed other LLM-based solutions by over 10% on average. |