Papers by Weiming Li
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debugging (2025.emnlp-main)
Copied to clipboard
Weiming Zhang, Qingyao Li, Xinyi Dai, Jizheng Chen, Kounianhua Du, Weiwen Liu, Yasheng Wang, Ruiming Tang, Yong Yu, Weinan Zhang
| Challenge: | Early debugging efforts focused on code-level analysis, which often fails when addressing complex programming errors. |
| Approach: | They propose a framework that employs natural language as an intermediate representation to improve code debugging by debuggating at a natural language level. |
| Outcome: | The proposed framework outperforms traditional debugging methods and enables a broader modification space through direct refinement guided by execution feedback. |
HiSA: Hierarchical State Abstraction for Scalable GUI Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent multimodal large language models (MLLMs) exploit insufficient state abstraction to automate workflows. |
| Approach: | They propose a hierarchical state abstraction approach that actively restructures knowledge rather than passively retaining historical information. |
| Outcome: | The proposed approach achieves a 40.58% success rate while reducing token consumption by 69.85% and monetary costs by 55.10% compared to the best-performing baseline. |
ToM-Synth: Scaling Robust Theory of Mind in LLMs via 6,912 Structured Social Units (2026.findings-acl)
Copied to clipboard
Guiyang Hou, Xiang Huang, Shangke Lyu, Yuchuan Wu, Weiyao Luo, Xinyu Mei, Yongliang Shen, Weiming Lu, Yongbin Li
| Challenge: | Existing methods endowing LLMs with Theory of Mind fail to internalize the augmented ToM into the LLM. |
| Approach: | They propose a factorial combinatorial synthesis framework that enables systematic synthesis of ToM data and uses it for RL fine-tuning. |
| Outcome: | The proposed framework yields a training dataset of 27,648 instances. |
Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization (2024.acl-long)
Copied to clipboard
Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, Weiming Lu
| Challenge: | Large Language Models (LLMs) are designed as specific task solvers with sophisticated prompt engineering, but are inherently incapacitating to address complex dynamic scenarios. |
| Approach: | They propose an LLM-based agent with policy-level reflection and optimization that can learn from interactive experiences and progressively elevate its behavioral policy. |
| Outcome: | The proposed agent outperforms vanilla LLM and specialized models in blackjack and Texas hold’em. |
CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System (2025.acl-long)
Copied to clipboard
Li Hu, Guoqiang Chen, Xiuwei Shang, Shaoyin Cheng, Benlong Wu, LiGangyang LiGangyang, Xu Zhu, Weiming Zhang, Nenghai Yu
| Challenge: | CompileAgent is the first LLM-based agent framework dedicated to repo-level compilation. |
| Approach: | They propose a LLM-based agent framework dedicated to repo-level compilation. |
| Outcome: | The proposed method significantly improves compilation success rate, ranging from 10% to 71%. |
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for video-text retrieval capture fine-grained semantic concepts . however, they lack the ability to capture finer-grain concepts such as objects and actions. |
| Approach: | They propose a dual-encoder architecture for fast video-text retrieval that learns lexicon representations to capture fine-grained semantics. |
| Outcome: | The proposed framework outperforms existing methods with 4.8% and 8.2% improvement on MSR-VTT and DiDeMo respectively. |
Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination mitigation are based on external dependency and require external annotations or auxiliary models for preference data collection. |
| Approach: | a new method is proposed to help model-generated hallucinations without external dependencies. |
| Outcome: | a new method that self-injects hallucinations into a generated response improves halluuutations mitigation. |
Unleashing the Power of LLMs in Court View Generation by Stimulating Internal Knowledge and Incorporating External Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made remarkable strides in language generation, but they encounter difficulties in the knowledge-intensive legal domain. |
| Approach: | They propose to decompose court views into different parts, stimulate internal knowledge, and incorporate external information to unleash the power of LLMs in the task. |
| Outcome: | The proposed method generates more accurate and reliable court views on two real-world datasets LAIC2021 and CJO2022. |
Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts (2026.acl-long)
Copied to clipboard
Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu, Yueting Zhuang
| Challenge: | Existing multimodal Mixture-of-Experts models accurately perceive image content yet fail in subsequent reasoning . Seeing but not thinking phenomenon is a puzzling phenomenon . |
| Approach: | They propose a routing-guided intervention method that enhances domain expert activation. |
| Outcome: | The proposed method achieves consistent improvements on visual reasoning tasks. |
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling. |
| Approach: | They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space. |
| Outcome: | The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks. |
Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation (2026.findings-acl)
Copied to clipboard
Zhiheng Li, Zongyang Ma, Yuntong Pan, Ziqi Zhang, Xiaolei Lv, Bo Li, Jun Gao, Jianing Zhang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Multimodal Large Language Models (MLLMs) are increasingly being deployed as content moderators . however, they exploit the Human-AI capability gap and create adversarial environments . smuggling attacks exploit the human-AI gap and exploit the vulnerability . |
| Approach: | They construct a benchmark to evaluate the vulnerability of MLLMs as content moderators . they identify three root causes: limited capabilities of vision encoders, robustness gap in OCR . |
| Outcome: | The proposed model exploits the Human-AI capability gap and is vulnerable to smuggling attacks. |
D-RAG: Differentiable Retrieval-Augmented Generation for Knowledge Graph Question Answering (2025.emnlp-main)
Copied to clipboard
Guangze Gao, Zixuan Li, Chunfeng Yuan, Jiawei Li, Wu Jianzhuo, Yuehao Zhang, Xiaolong Jin, Bing Li, Weiming Hu
| Challenge: | Existing approaches to Knowledge Graph Question Answering (KGQA) use Retrieval-Augmented Generation (RAG) but subgraph selection process is non-differentiable, preventing end-to-end training of the retriever and the generator. |
| Approach: | They propose a Differentiable RAG approach that optimizes the retriever and the generator for KGQA. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches on WebQSP and CWQ. |
LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). |
| Approach: | They propose a training-free method that remaps out-of-distribution (OOD) positions into the in-distance range with fixed mapping strategies, ignoring the dynamic relationship between input length and effective context window. |
| Outcome: | Experiments on three representative LLMs across five mainstream long-context benchmarks show that the proposed method achieves significant performance improvements compared to existing methods. |
DiffusionNER: Boundary Diffusion for Named Entity Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks are fundamental to many structured information extraction tasks. |
| Approach: | They propose a named entity recognition task that uses a boundary-denoising diffusion process to denoise noisy spans. |
| Outcome: | The proposed method achieves comparable or even better performance than previous state-of-the-art models on flat and nested datasets. |
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing cross-lingual transfer are limited by parallel resources and lack linguistic and domain coverage. |
| Approach: | They propose a cross-lingual in-context pre-training approach that leverages semantically related bilingual Wikipedia documents to enhance cross-linguistic transfer. |
| Outcome: | The proposed approach improves multilingual performance on three models across six target languages. |
Legal Judgment Prediction based on Knowledge-enhanced Multi-Task and Multi-Label Text Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Legal judgment prediction (LJP) is an essential task for legal AI, aiming at predicting judgments based on the facts of a case. |
| Approach: | They propose a knowledge-enhanced approach that incorporates 'label-level knowledge' to enhance the representation of case facts for each task and 'task-level' knowledge to improve synergy. |
| Outcome: | The proposed method is effective in comparison to state-of-the-art (SOTA) baselines. |
Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks (2026.acl-long)
Copied to clipboard
Wenqi Zhang, Mengna Wang, Gangao Liu, Huixin Xu, Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, Jiajun Liu, Weiming Lu, Peng Li, Yueting Zhuang
| Challenge: | Recent advances in reasoning models have demonstrated remarkable capabilities on mathematical and coding tasks, but their effectiveness in embodied domains remains largely unexplored. |
| Approach: | They propose a reasoning model for interactive embodied tasks that synthesizes 9.3k coherent Observation-Thought-Action trajectories containing 64k ego-centric images and 90k diverse reasoning processes. |
| Outcome: | The proposed model outperforms existing visual reasoning models by +9%, 24%, and +13% on long-horizon tasks. |
Into the Gray Zone: Domain Contexts Can Blur LLM Safety Boundaries (2026.acl-long)
Copied to clipboard
Ki Sen Hung, Xi Yang, Chang Liu, Haoran Li, Kejiang Chen, Changxuan Fan, Tsun On Kwok, Weiming Zhang, Xiaomeng Li, Yangqiu Song
| Challenge: | a goal of LLM alignment is to balance usefulness with harmlessness, but this conflictes when knowledge serves both legitimate and malicious purposes. |
| Approach: | They propose a framework that combines safety-research contexts with adversarial interactions to exploit a vulnerability in Jargon queries. |
| Outcome: | a framework outperforms existing methods in analyzing Jargon queries, a study shows . it achieves 93% of attacks across seven models, while remaining useful, the authors say . |
AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs (2026.acl-long)
Copied to clipboard
Qingqing Lyu, Linjuan Wu, Yongliang Shen, Hengwei Liu, Hao Li, Shengpei Jiang, Yin Zhang, Weiming Lu
| Challenge: | Existing automated approaches operate within fixed task schemas and often fail to autonomously discover new evaluation dimensions. |
| Approach: | They propose an automated framework that constructs domain-specific benchmarks directly from unstructured corpora using Bloom’s Taxonomy. |
| Outcome: | The proposed framework uncovers a broader and more fine-grained task space than expert-curated benchmarks while producing high-quality instances that preserve established model-level evaluation trends. |