Papers by Austin Xu
Direct Judgement Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing judge models are largely trained with supervised finetuning on small data scales to perform limited types of evaluation tasks, limiting generalization. |
| Approach: | They propose to train judge models at large data scales with direct preference optimization . they use four training tasks to form three types of preference pairs targeting different aspects of evaluation . |
| Outcome: | The proposed model outperforms GPT-4o and other similar models on 13 benchmarks. |
J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly being used for reasoning intensive tasks. |
| Approach: | They propose an algorithm that trains judges to be robust to positional biases . they also propose a benchmark that evaluates judges in diverse reasoning settings . |
| Outcome: | The proposed algorithm outperforms GPT-4o and the next best small judge by 6.7% and 9% on ReasoningJudgeBench and JudgeBench. |
Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive results in single-document summarization, but their performance on MDS still leaves room for improvement. |
| Approach: | They propose a topic-guided reinforcement learning approach to improve content selection in MDS . explicit prompting models with topic labels enhances the informativeness, they show . |
| Outcome: | The proposed method outperforms baselines on multi-News and multi-XScience datasets. |
Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math (2026.acl-long)
Copied to clipboard
| Challenge: | Large language model (LLM)-based reasoning systems have recently achieved gold medal-level performance in the IMO 2025 competition . |
| Approach: | They propose a human-annotated step-level verification benchmark that measures step- level verifiers at the frontier. |
| Outcome: | The proposed benchmark outperforms closed-source models in step-level verification and the impact of scaling verifier compute. |
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings (2025.acl-long)
Copied to clipboard
| Challenge: | Contextual evaluation is challenging for state-of-the-art judge models . evaluation criteria are often conditional and dependent on practitioner priorities . |
| Approach: | They propose a judge benchmark that evaluates large language models as judges in contexts . they use human annotations and model-based perturbations to build the benchmark . |
| Outcome: | The proposed benchmark aims to evaluate large language models in contexts with 2,000 challenging response pairs. |