Papers by Haoyuan Hu
Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Recent Large Multimodal Models (LMMs) focus on visual knowledge-dimension alignment, but ignore visual knowledge. |
| Approach: | They propose a cognitive visual-language mapper that integrates visual-linguistic knowledge alignment with a fine-grained knowledge Adapter. |
| Outcome: | The proposed model significantly improves LMMs on knowledge-based visual question answering (VQA) it also improves the performance of other models, including GPT-4V and Gemini-Pro. |
Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction (2024.acl-long)
Copied to clipboard
| Challenge: | Recent research on temporal fact extraction fails to establish time-to-fact correspondences in complex sentences. |
| Approach: | They propose a timeline-based sentence decomposition strategy using large language models with in-context learning to extract temporal facts from natural language text. |
| Outcome: | The proposed method achieves state-of-the-art on a complex temporal fact extraction dataset. |
Leveraging Outline-Optimized Generative Interactions and Critique for Self-Refining Outlines with Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Logic-RL is a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Approach: | They propose a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Outcome: | The proposed framework improves on FreshWiki and WikiOutline . it can be iteratively applied, with improved quality continuing through three refinement rounds before diminishing returns. |
MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for complex multi-shot video are anchored to single-shot paradigms, lacking comprehensive story assets and cross-shot metrics. |
| Approach: | They propose a framework that synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models. |
| Outcome: | The proposed framework synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models. |
UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity Mixture-of-Experts (2026.acl-long)
Copied to clipboard
Zhenyu Liu, Yunxin li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Haolan Chen, Fanbo Meng, Mingjun Zhao, Yu Xu, Yancheng He, Baotian Hu, Haizhou Li, Min Zhang
| Challenge: | Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. |
| Approach: | They propose a unified speech and music generation model built upon a novel framework . they propose specialized MoE architectures and curated training strategies to tackle data imbalances . |
| Outcome: | The proposed model achieves state-of-the-art performance on major speech and music generation benchmarks. |
VideoVista-CulturalLingo: 360° Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension (2025.acl-long)
Copied to clipboard
| Challenge: | Existing video evaluation benchmarks focus on a single language, typically English, and feature videos rooted in Western cultural contexts. |
| Approach: | They propose a video evaluation benchmark designed to bridge cultural, linguistic, and domain divide in video comprehension. |
| Outcome: | The proposed video evaluation benchmark bridges cultural, linguistic, and domain divides . existing benchmarks only feature videos from YouTube, Shutterstock, or established video datasets based on cultural diversity . |
Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering (2024.lrec-main)
Copied to clipboard
Yuan Gao, Yiheng Zhu, Yuanbin Cao, Yinzhi Zhou, Zhen Wu, Yujie Chen, Shenglan Wu, Haoyuan Hu, Xinyu Dai
| Challenge: | Open Domain Multi-Hop Question Answering (ODMHQA) is one of the most challenging tasks in Natural Language Processing (NLP) |
| Approach: | They propose a mechanism that leverages the intrinsic capabilities of Large Language Models to judge whether the generated answers are off-topic. |
| Outcome: | The proposed method reduces the occurrence of off-topic answers by nearly 13%, improving the performance in Exact Match (EM) by nearly 3% compared to the baseline method without the Dr3 mechanism. |