| Challenge: | Large Language Models (LLMs) can exhibit considerable variation in quality of sampled outputs. |
| Approach: | They propose a method for reranking LLM generations using pairwise statistics . they show strong improvements for selecting the best k generations for code generation tasks . |
| Outcome: | The proposed approach improves selection and generation quality for code generation tasks and autoformalization, summarization, and translation tasks. |
Similar Papers
FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing listwise LLMs lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. |
| Approach: | They propose a listwise LLM reranking approach that leverages the first generated identifier to obtain a ranked ordering of the candidates. |
| Outcome: | The proposed approach accelerates inference by 50% while maintaining robust ranking performance with gains across BEIR benchmark. |
Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for large language models overlook the inherent randomness of LLMs. |
| Approach: | They propose a hierarchical statistical model that incorporates both benchmark characteristics and LLM randomness to provide a more comprehensive representation of benchmarking process. |
| Outcome: | The proposed model improves the accuracy of estimating the benchmark score and reduces variance. |
Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate relevance labels for large language models have not been successful in generating relevance labels. |
| Approach: | They propose a method to combine LLM relevance labels with ranking abilities . they take both LLM generated relevance labels and pairwise preferences . |
| Outcome: | The proposed method balances the ranking and labeling abilities of large language models . it takes both LLM generated relevance labels and pairwise preferences . |
URG: A Unified Ranking and Generation Method for Ensembling Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to rank and generate large language models have limited performance due to time-intensive nature of ranking process and lack of error propagation. |
| Approach: | They propose a framework that jointly ranks the outputs of Large Language Models and generates fine-grained fusion results. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) performance on ranking and generation tasks. |
Large Language Models for Generative Recommendation: A Survey and Visionary Discussions (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing but are not fully able to leverage the generative power of LLM. |
| Approach: | They examine the progress, methods, and future directions of large language models . they examine what generative recommendation is, why RS should advance to generative recommendations . |
| Outcome: | The proposed approach can be simplified to generate recommendations from the entire pool of items. |
REQUAL-LM: Reliability and Equity through Aggregation in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new method for finding reliable and equitable LLM outputs is developed . REQUAL-LM does not require specialized hardware and does not impose a significant computing load . |
| Approach: | They propose a method for finding reliable and equitable LLM outputs through aggregation. |
| Outcome: | The proposed method minimizes harmful bias while finding reliable outputs . it does not require specialized hardware and does not impose a significant computing load . |
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation (2023.findings-acl)
Copied to clipboard
Xingdi Yuan, Tong Wang, Yen-Hsiang Wang, Emery Fine, Rania Abdelghani, Hélène Sauzéon, Pierre-Yves Oudeyer
| Challenge: | Large Language Models (LLMs) have demonstrated impressive prowess in natural language generation. |
| Approach: | They propose a method to select high-quality questions from LLM-generated candidates using round-trip and prompt-based scoring. |
| Outcome: | The proposed approach can select high-quality questions from a set of LLM-generated candidates without modification of the underlying model nor rely on human annotations. |
A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to generate synthetic textual data for training smaller specialized models. |
| Approach: | They evaluate the performance of large language models and their generation strategies in 11 different languages using 3 NLP tasks and 4 open-source LLMs. |
| Outcome: | The proposed generation strategies and their combinations yield strong results across 11 languages, including several extremely low-resource ones. |
Improving LLM Generations via Fine-Grained Self-Endorsement (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable capabilities but can still fail frequently on knowledge-intensive tasks. |
| Approach: | They propose a self-endorsement framework that leverages fine-grained fact-level comparisons across multiple sampled responses. |
| Outcome: | The proposed framework can improve factuality of generations with simple prompts across scales of LLMs. |
Make Large Language Model a Better Ranker (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate robust capabilities across various fields . current list-wise approaches fail in ranking tasks due to misalignment between ranking objectives and next-token prediction . |
| Approach: | They propose a large language model framework with Aligned Listwise Ranking Objectives (ALRO) this framework provides explicit feedback in a listwise manner by introducing soft lambda loss . |
| Outcome: | The proposed model outperforms existing recommendation methods and embedding-based recommendations without additional computational burdens. |