Papers by Yibo Hu
Controllable Fake Document Infilling for Cyber Deception (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to deter malicious intrusion generate multiple fake versions of a document that are context-agnostic and produce sub-optimal outputs. |
| Approach: | They propose a context-aware model for creating fake documents that masks important concepts of varied lengths and then infills a realistic but fake alternative considering both the previous and future contexts. |
| Outcome: | The proposed model outperforms baseline models in generating highly believable fakes with moderate modification to protect critical information and deceive adversaries. |
Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla P Gomes, Bart Selman, Qingsong Wen
| Challenge: | Current scientific reasoning models struggle with generalization across domains and fall short of multimodal perception. |
| Approach: | They propose to use multimodal large language models to integrate text, images, and other modalities to enhance scientific reasoning. |
| Outcome: | The proposed models can integrate text, images, and other modalities and improve reasoning across disciplines. |
VLA-Mark: A cross modal watermark for large vision-language alignment models (2025.emnlp-main)
Copied to clipboard
Shuliang Liu, Zheng Qi, Jesse Jiaxi Xu, Yibo Yan, Junyan Zhang, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu
| Challenge: | Existing text watermarking methods disrupt visual-textual alignment, leaving semantic-critical concepts vulnerable. |
| Approach: | They propose a vision-aligned framework that embeds detectable watermarks into outputs . they combine localized patch affinity, global semantic coherence, contextual attention patterns . |
| Outcome: | The proposed framework shows lower PPL and higher BLEU than conventional methods with near-perfect detection (98.8% AUC). |
PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions (2025.findings-emnlp)
Copied to clipboard
Song Dai, Yibo Yan, Jiamin Su, Zihao Dongfang, Yubo Gao, Yonghua Hei, Jungang Li, Junyan Zhang, Sicheng Tao, Zhuoran Gao, Xuming Hu
| Challenge: | Current physics benchmarks focus on text-only inputs or only on problem-solving . current physics reasoning benchmarks neglect critical intermediate steps of variable identification and process formulation. |
| Approach: | a new benchmark evaluates multimodal large language models in physics reasoning . the benchmark measures variables, process formulations, and solution derivation . |
| Outcome: | PhysicsArena is the first multimodal physics reasoning benchmark . it evaluates MLLMs across three critical dimensions: variable identification, process formulation, and solution derivation. |
Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by hallucinations, a particularly challenging variant, knowledge overshadowing, which can lead to erroneous outputs even with high-quality training data. |
| Approach: | They propose a framework to analyze and detect knowledge overshadowing by using knowledge circuit analysis to dissect the function of key components in the circuit and how attention pattern dynamics contribute to the phenomenon. |
| Outcome: | Extensive experiments show that the framework can detect and analyze knowledge overshadowing and improves on existing models. |
ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence (2022.naacl-main)
Copied to clipboard
Yibo Hu, MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, Vito D’Orazio
| Challenge: | Traditionally, researchers used manual coding to track conflict processes worldwide, but the high costs and slow pace of domain experts make it difficult and costly to monitor complex and rapidly changing conflicts. |
| Approach: | They propose a domain-specific pre-trained language model for conflict and political violence that can be used to train a language model from scratch and continue training. |
| Outcome: | The proposed model outperforms BERT in conflict research. |
ChatEdit: Towards Multi-turn Interactive Facial Image Editing via Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to interactive facial image editing treat multi-turn editing as a sequence of successive single-turn edits, leading to attribute forgetting and error accumulation. |
| Approach: | They propose a framework for interactive facial image editing through dialogues based on the CelebA-HQ dataset and a benchmark dataset to evaluate this. |
| Outcome: | The proposed framework outperforms existing methods and improves existing ones. |
ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, YiFan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu, Qingsong Wen
| Challenge: | Current mathematical benchmarks focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection. |
| Approach: | They propose to evaluate multimodal error detection by evaluating two sub-tasks error step identification and error categorization. |
| Outcome: | The proposed task evaluates MLLMs' ability to handle multimodal questions compared to text-only models. |
Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification (2024.acl-long)
Copied to clipboard
| Challenge: | Existing annotation codebook is labor-intensive for coding events from large datasets. |
| Approach: | They propose to use existing annotation codebook to classify political relations without extensive annotations. |
| Outcome: | The proposed methods outperform dictionary-based methods and the existing ontology annotation codebook and improve interpretability and efficiency. |
LLM Agents for Education: Advances and Applications (2025.findings-emnlp)
Copied to clipboard
Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jingheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S. Yu, Qingsong Wen
| Challenge: | Large Language Model (LLM) agents are transforming education by automating complex tasks and enhancing both teaching and learning processes. |
| Approach: | This survey analyzes recent advances in applying Large Language Model agents to educational settings . it highlights ethical issues, hallucination and overreliance, and integration with existing ecosystems . |
| Outcome: | The authors analyze the technologies enabling LLM agents and highlight key challenges in deploying them in educational settings. |
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models (2025.findings-acl)
Copied to clipboard
Jiamin Su, Yibo Yan, Fangteng Fu, Zhang Han, Jingheng Ye, Xiang Liu, Jiahao Huo, Huiyu Zhou, Xuming Hu
| Challenge: | Automated Essay Scoring (AES) systems face three major challenges: reliance on handcrafted features that limit generalizability, difficulty in capturing fine-grained traits like coherence and argumentation, and inability to handle multimodal contexts. |
| Approach: | They propose a multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering. |
| Outcome: | The proposed system can evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering. |
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios (2025.emnlp-main)
Copied to clipboard
Yunkai Dang, Mengxi Gao, Yibo Yan, Xin Zou, Yanggan Gu, Jungang Li, Jingyu Wang, Peijie Jiang, Aiwei Liu, Jia Liu, Xuming Hu
| Challenge: | Existing studies have focused mainly on visual–textual misalignment, leaving largely unexplored the MLLMs’ ability to preserve an original correct answer when confronted with misleading information. |
| Approach: | They propose a two-stage evaluation pipeline to quantify the response uncertainty phenomenon by eliciting each model’s original response on unperturbed inputs and injecting explicit (false-answer hints) and implicit (contextual contradictions) misleading instructions. |
| Outcome: | The proposed model overturns a correct answer in 65% of cases after receiving a single deceptive cue. |
Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs (2026.findings-acl)
Copied to clipboard
Yubo Gao, Haotian Wu, Hong Chen, Junquan Huang, Yibo Yan, Jungang Li, Zihao Dongfang, Sicheng Tao, PS Tan, Jie Zhang, Xuming Hu
| Challenge: | Existing efficiency methods for Chain-of-Thought (CoT) generate excessively long rationales without commensurate accuracy gains. |
| Approach: | They propose a training framework that operationalizes this principle through coarse-to-fine budgeting. |
| Outcome: | Experiments on GSM8K and MATH500 show that HAB surpasses standard CoT in accuracy and reduces token usage, achieving stronger performance-efficiency trade-off than baselines. |
MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing MLLMs have a visual question answering capability but lack domain-specific information. |
| Approach: | They propose a framework for language model modules in MLLMs when handling projected image features and verify this hypothesis using logit lens. |
| Outcome: | The proposed framework will yield a 10% change in accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future. |
Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering (2026.acl-long)
Copied to clipboard
Shuliang Liu, Songbo Yang, Dong Fang, Sihang Jia, Yuqi Tang, Lingfeng Su, Ruoshui Peng, Yibo Yan, Xin Zou, Xuming Hu
| Challenge: | Existing approaches to overcome object hallucination are limited . Existing mitigations include costly retraining and a training-free inference framework . |
| Approach: | They propose a training-free inference framework that simulates a metacognitive self-correction process. |
| Outcome: | The proposed framework reduces object hallucination rates by 12.67% on MMHal-Bench and improves accuracy by 5.8% on POPE. |
Beyond ’Aha!’: Toward Systematic Meta-Abilities Alignment in Large Reasoning Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work has shown that outcome-based reinforcement learning (RL) can incidentally elicit advanced reasoning behaviors such as self-correction, backtracking, and verification. |
| Approach: | They explicitly align large reasoning models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks. |
| Outcome: | The proposed model aligns models with deduction, induction, and abduction meta-abilities using automatically generated, self-verifiable tasks. |
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) is of importance in multimodal retrieval applications. |
| Approach: | They propose a two-stage pruning and merging frameworks that combine pruning and merge techniques to achieve higher compression rates. |
| Outcome: | The proposed framework outperforms existing methods on 29 visual document retrieval datasets. |
Detecting AI-Generated Content on Social Media with Multi-modal Language Models (2026.acl-industry)
Copied to clipboard
Chenyang Yang, Shen Yan, Yibo Yang, Litao Hu, Yuchen Liu, Yuan Zeng, Hanchao Yu, Yinan Zhu, Sumedha Singla, Brian Vanover, Huijun Qian, Zihao Wang, Fujun Liu, Aashu Singh, Jianyu Wang, Xuewen Zhang
| Challenge: | Existing methods for AI-generated content detection face poor generalization to newer models, reliance on single modalities, and lack of interpretable explanations. |
| Approach: | They propose a model that curates diverse social media data and trains a vision-language model for detection and explanation. |
| Outcome: | The proposed model achieves state-of-the-art detection performance on public benchmarks and observes positive downstream impacts on user engagement. |
A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models (2025.findings-acl)
Copied to clipboard
Shuliang Liu, Hongyi Liu, Aiwei Liu, Duan Bingchen, Zheng Qi, Yibo Yan, He Geng, Peijie Jiang, Jia Liu, Xuming Hu
| Challenge: | Existing methods for detection of misinformation generated by large language models fail to mitigate societal risks . authors propose a paradigm shift from passive detection to anticipatory mitigation strategies . existing defenses remain reactionary in an era demanding proactive defense, authors say . |
| Approach: | They propose a three-pillar approach to prevent misinformation by fortifying integrity of training data and inference reliability by embedding self-corrective mechanisms during reasoning. |
| Outcome: | The proposed framework improves existing methods in misinformation prevention by 63% . it demonstrates that existing methods exhibit false negative rates against misinformation . |
A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges (2025.findings-acl)
Copied to clipboard
Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu
| Challenge: | This survey provides **the first comprehensive analysis of mathematical reasoning in the era of multimodal large language models** . integrating large language model with mathematical reasoning tasks is becoming significant as AI advances . |
| Approach: | They review over 200 studies published since 2021 and examine the state-of-the-art developments in Math-LLMs . they identify five major challenges hindering the realization of AGI in this domain . |
| Outcome: | The authors examine the state-of-the-art developments in Math-LLMs with a focus on multimodal settings. |
MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection (2025.acl-industry)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) struggle with identifying and categorizing student errors in multimodal mathematical contexts. |
| Approach: | They propose a new framework that decomposes error detection into three phases with specialized agents. |
| Outcome: | The proposed framework shows higher accuracy in error step identification and 3% improvement in error categorization on real-world educational data. |
SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning (2025.findings-acl)
Copied to clipboard
Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, Xuming Hu
| Challenge: | Existing methods for MU forget quality and model utility are not fully explored for safety in MLLMs. |
| Approach: | They propose a safety unlearning benchmark for MLLMs to measure over-forgetting . they propose MU methods to forget quality and model utility . |
| Outcome: | The proposed method reduces over-forgetting by 79.5% while maintaining forget quality and model utility. |
MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in machine learning (MU) have enabled the selective removal of private or sensitive information encoded within deep neural networks. |
| Approach: | They propose to "reformulate" the task of multimodal MU in the era of MLLMs by preserving only the visual patterns associated with a given entity while preserving the corresponding textual knowledge. |
| Outcome: | The proposed method surpasses baselines that finetuned MLLMs with VQA data directly through Gradient Ascent (GA) or Negative Preference Optimization (NPO), across all evaluation dimensions. |
Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research focuses on object-level or attribute-level hallucinations, neglecting the more complex relation hallucinosities. |
| Approach: | They propose a comprehensive benchmark targeting relation hallucinations comprising over 20,000 real-world samples and a confidence-based mitigation strategy which reduces the halluciation rate by an average of 9.75% across three datasets. |
| Outcome: | The proposed approach reduces the hallucination rate by an average of 9.75% across three datasets, including Reefknot. |
Unlocking Speech Instruction Data Potential with Query Rewriting (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack datasets and biased training tasks to follow speech instructions. |
| Approach: | They propose a query rewriting framework that uses multiple agents to annotate and validate the synthesized speech. |
| Outcome: | The proposed framework can transform text instructions into distributions more suitable for TTS models for speech synthesis without human annotation. |
Scaling Law for Multimodal Large Language Model Supervised Fine-Tuning (2026.acl-long)
Copied to clipboard
YiFan Zhang, Tao Yu, Feng Li, Chaoyou Fu, Yibo Hu, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin
| Challenge: | supervised fine-tuning (SFT) is crucial for multimodal large language models, yet a comprehensive scaling law is lacking . et al.: scaling laws focus on model size, pre-training tokens, and MLLM SFT data volumes . |
| Approach: | They propose two scaling laws to guide optimal model-data configuration . they propose one applicable when training data volumes are well defined by researchers . |
| Outcome: | The proposed scaling laws provide valuable recommendations for optimal resource allocation . they show that the proposed laws are more accurate than existing models . |