Using tournaments to calculate AUROC for zero-shot classification with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models perform well on zero-shot classification tasks, but are difficult to compare to supervised classifiers due to the lack of a modifiable decision boundary. |
| Approach: | They propose a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset. |
| Outcome: | The proposed method produces relative rankings of instances using the Elo rating system . |
Similar Papers
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled impressive zero-shot capabilities across various natural language tasks. |
| Approach: | They propose two ways to exploit the emergent abilities of large language models for NLG assessment. |
| Outcome: | The proposed methods improve performance and positional biases in comparisons between candidates. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to automate essay scoring rely on LLMs to generate absolute scores . however, these methods diverge from human evaluations due to model biases and inconsistent scoring . |
| Approach: | They propose a method that formulates AES as a pairwise comparison task using large language models. |
| Outcome: | The proposed method outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. |
Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2024.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive zero-shot capabilities in various passage ranking tasks. |
| Approach: | They analyze and compare the effectiveness of monolingual reranking using query or document translations and evaluate the effectiveness when leveraging their own generated translations. |
| Outcome: | The proposed models perform better when using their own translations than when using query or document translations. |
Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. |
| Approach: | They propose to make Large Language Models (LLMs) operating in 0-shot or few-shot settings as efficient as 0- shot text classifiers by leveraging a small number of samples. |
| Outcome: | The proposed model is able to perform better on multiple datasets than existing models on 0-shot or few-shot settings. |
A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating CNs are expensive, time-consuming, and subjective, but lack a universal truth and the lack of a 'universal truth' . |
| Approach: | They propose a model ranking pipeline based on pairwise comparisons of generated CNs from different models organized in a tournament-style format to improve the evaluation process. |
| Outcome: | The proposed method achieves a high correlation with human preference, with a score of 0.88, and compares chat, instruct, and base models, exploring their strengths and limitations. |
Finetuning LLMs for Comparative Assessment Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Automated assessment in natural language generation is a challenging task. |
| Approach: | They propose a framework for fine-tuning LLMs for comparative assessment to align the model’s output with the target distribution of comparative probabilities. |
| Outcome: | The proposed framework improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons. |
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods is presented. |
| Approach: | They evaluate 22 reranking methods including 40 variants across established benchmarks . primary goal is to determine whether performance disparity exists between LLM-based reranters and lightweight counterparts based on novel queries . |
| Outcome: | The proposed methods perform better on familiar queries than lightweight models, the authors show . |
BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination (2026.acl-long)
Copied to clipboard
| Challenge: | Existing lists of document ranking methods lack robust performance across domains. |
| Approach: | They propose a reasoning-driven competitive elimination framework that optimises group sizes based on LLM context limits and reasoning-enhanced prompts. |
| Outcome: | The proposed method outperforms RankGPT and other state-of-the-art methods on datasets with a 77.90 NDCG@5 score and 54.66 average NDGC@10 on BEIR datasets. |
Aligning Black-box Language Models with Human Judgments (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as automated judges to evaluate recommendation systems, search engines, and other subjective tasks. |
| Approach: | They propose a framework to align LLM judgments with individual human evaluators or their aggregated judgments without retraining or fine-tuning the LLM. |
| Outcome: | The proposed framework achieves 142% improvement in agreement across 29 tasks and exceeds inter-human agreement on four out of six tasks. |