Papers by Guolin Huang
SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis (2025.findings-naacl)
Copied to clipboard
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Wang Changxin, Zhifeng Gao, Hongshuai Wang, Li Yongge, Mujie Lin, Shuwen Yang, Jiankun Wang, Mingjun Xu, Jin Huang, Xi Fang, Jiaxi Zhuang, Yuqi Yin, Yaqi Li, Changhong Chen, Zheng Cheng, Zifeng Zhao, Linfeng Zhang, Guolin Ke
| Challenge: | Existing benchmarks fail to adequately evaluate the proficiency of Large Language Models (LLMs) Existing standards do not cover the skills needed to evaluate LLMs in scientific literature analysis. |
| Approach: | They propose a benchmark to evaluate the proficiency of large language models in scientific literature analysis. |
| Outcome: | SciAssess evaluates 11 LLMs on multiple tasks across scientific fields. |
MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing medical benchmarks fail to detect the Einstellung Effect in clinical diagnosis . Existing models exhibit the Einstellung effect, relying on statistical shortcuts rather than logical reasoning. |
| Approach: | They propose a counterfactual benchmark that uses statistical shortcuts to diagnose patients . they propose CGME-based system that iteratively refines reasoning paths . |
| Outcome: | The proposed model achieves high baseline accuracy but severe bias trap rates . iteratively refines reasoning paths in an exemplar base and consolidates disease-specific knowledge into illness graphs. |