Papers by Zhiyi Song
Laying the Groundwork for Knowledge Base Population: Nine Years of Linguistic Resources for TAC KBP (L18-1)
Copied to clipboard
| Challenge: | Knowledge Base Population (KBP) evaluations target information extraction technologies for knowledge bases comprised of entities, relations, and events. |
| Approach: | They describe the linguistic resources provided by Linguistic Data Consortium for TAC KBP since 2009 . they highlight changes made to support evolving evaluation requirements . |
| Outcome: | The evaluations have targeted information extraction technologies for the population of knowledge bases comprised of entities, relations, and events. |
Cross-Document, Cross-Language Event Coreference Annotation Using Event Hoppers (L18-1)
Copied to clipboard
| Challenge: | Defined event hoppers for the DEFT Rich Entities, Relations and Events (Rich ERE) annotation task. |
| Approach: | They propose an approach for cross-document, cross-lingual event coreference for the DEFT Rich Entities, Relations and Events (Rich ERE) annotation task. |
| Outcome: | The proposed approach is based on the definition of event hoppers for the DEFT rich entities, relations, events and their attributes . it yields 389 cross-document event hoppings in 505 documents in three languages . |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
CMB: A Comprehensive Medical Benchmark in Chinese (2024.naacl-long)
Copied to clipboard
Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, Haizhou Li
| Challenge: | Large Language Models (LLMs) provide a great breakthrough in medicine, says a new study . existing studies on LLMs leverage subjective evaluation, but evaluation in medicine is professional . |
| Approach: | They propose a localized medical benchmark in Chinese rooted in native Chinese . they propose to use traditional Chinese medicine to evaluate large-scale LLMs . |
| Outcome: | a new benchmark is developed to evaluate large-scale LLMs in china . the proposed model is rooted in the native Chinese linguistic and cultural framework . |