Papers by Jiulong Shan
MR. Judge: Multimodal Reasoner as a Judge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective reward modeling is especially valuable in reinforcement learning (RLHF) . |
| Approach: | They propose a paradigm for empowering general-purpose MLLMs judges with strong reasoning capabilities by using multiple-choice problem models instead of directly assigning scores. |
| Outcome: | The proposed model surpasses GPT-4o on VL-RewardBench and improves performance on MM-Vet by up to 7.7%. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate preference data without human annotations are difficult . et al., 2022b) is effective for aligning large language models with human expectations . |
| Approach: | They propose a method to evaluate the response preference using output probabilities under contrastive prompts. |
| Outcome: | The proposed method could surpass the RLHF method without human-annotated preference data. |