Papers by Shibo Hao
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models (2026.acl-long)
Copied to clipboard
Yanbin Yin, Kun Zhou, Zhen Wang, Xiangdong Zhang, Yifei Shao, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin, Zhiting Hu
| Challenge: | closed-ended question-based benchmarks struggle with saturation as newer models emerge . crowd-sourced leaderboards rely on costly and slow human judges . |
| Approach: | They propose a framework that leverages collective intelligence from all large language models to evaluate each other. |
| Outcome: | a new framework enables a democratic, pairwise evaluation of all large language models . it achieves 97% correlation with human judgements, while significantly reducing the cost. |
Reasoning with Language Model is Planning with World Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable reasoning capabilities, particularly with Chain-of-Thought-style prompts. |
| Approach: | They propose a framework that repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) |
| Outcome: | The proposed framework repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) it achieves optimum balance between exploration and exploitation, while achieving high-reward reasoning paths efficiently. |
Offline Reinforcement Learning for LLM Multi-step Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly applied to complex tasks requiring multi-step reasoning. |
| Approach: | They propose an offline method for enhancing multi-step reasoning by optimizing the soft Bellman Equation by combining a policy model and a value function. |
| Outcome: | The proposed method surpasses existing methods on multi-step reasoning benchmarks and can be extended to multi-iteration frameworks when additional resources are available. |
DIDS: Domain Impact-aware Data Sampling for Large Language Model Training (2025.emnlp-main)
Copied to clipboard
Weijie Shi, Jipeng Zhang, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Yao Zhao, Hao Chen, Ruiyuan Zhang, Yue Cui, Jia Zhu, Sirui Han, Jiajie Xu, Xiaofang Zhou
| Challenge: | Existing approaches for optimizing domain-level sampling strategies struggle with maintaining intra-domain consistency and accurately measuring domain impact. |
| Approach: | They propose to use a Fisher-Information Matrix-guided metric to measure domain impact to ensure intra-domain consistency and accuracy. |
| Outcome: | The proposed model achieves 3.4% higher average performance while maintaining comparable training efficiency. |
Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). |
| Approach: | They propose a reasoning over commonsense knowledge bases (CSKBs) that are free-text and have a human annotation set to probe commonsensical reasoning. |
| Outcome: | The proposed model is based on a human-annotated evaluation set and is compared with existing models on the population task. |
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark (2026.findings-acl)
Copied to clipboard
Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Yaozhen Liang, WenHao Wang, Siheng Chen, Zhengxi Lu, Shuai Ren, Hao Wang, Shibo He, Yong Liu, Wenchao Meng
| Challenge: | Mobile GUI agents show promise in automating tasks but face significant generalization challenges in long-tail scenarios. |
| Approach: | They propose a benchmark framework for mobile GUI agents that measures the performance of GUI agents by analyzing their performance. |
| Outcome: | The LearnGUI benchmark outperforms existing methods in offline and online evaluations and demonstrates consistent gains across model architectures. |
BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to construct knowledge graphs are limited to a small set of relations due to manual cost or restrictions in text corpus. |
| Approach: | They propose to automatically construct knowledge graphs (KGs) of diverse new relations from pretrained language models that accept knowledge queries with prompts. |
| Outcome: | The proposed framework extracts knowledge of over 400 new relations from pretrained language models, including RoBERTaNet, with minimal input of a relation definition and a few shot of example entity pairs. |
Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction is a challenging task as it requires reasoning across multiple sentences. |
| Approach: | They propose to use a recommend-revise scheme to reduce the workload of annotators by providing them with candidate relation instances from distant supervision to supplement and remove relational facts. |
| Outcome: | The proposed dataset is the first large-scale and human-annotated dataset for relation extraction. |