Papers by Haiyang Zhang

How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States (2024.findings-emnlp)

Challenge:	Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored.
Approach:	They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios.
Outcome:	The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC.

Copied to clipboard

Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Yongbin Li

IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (2025.acl-long)

Challenge:	Large language models (LLMs) rely on safety alignment to avoid malicious user inputs.
Approach:	They employ weak classifiers to explain LLM safety through the intermediate hidden states.
Outcome:	The proposed model can identify malicious and normal inputs and detect malicious ones without jailbreak.

Copied to clipboard

Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, Yongbin Li

Challenge:	Existing algorithms to improve the ability of LLMs to follow complex instructions are lacking.
Approach:	They propose a benchmark to improve the ability to follow complex instructions by using a IOPO alignment method to take input and output preference into consideration.
Outcome:	The proposed algorithm shows 8.15%, 2.18% improvements on in-domain data and 5.91%, 2.83% on out-of-domain datasets compared to SFT and DPO respectively.

Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023.acl-long)

Copied to clipboard

Chaoya Jiang, Wei Ye, Haiyang Xu, Songfang Huang, Fei Huang, Shikun Zhang

mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding (2024.findings-emnlp)

Challenge:	Large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks.
Approach:	They propose a new framework for a semantically-aware contrastive learning that minimizes the MI between false negative and positive samples .
Outcome:	The proposed framework minimizes the MI between false negative samples and positive samples even though they share similar semantics.

Copied to clipboard

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou

mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding (2025.acl-long)

Challenge:	Existing Multimodal Large Language Models lack general structure understanding abilities for text-rich document images.
Approach:	They propose to use unified structure learning to boost the performance of MLLMs by encoding structure information into text-rich images.
Outcome:	The proposed model achieves state-of-the-art on 10 visual document understanding benchmarks.

Copied to clipboard

Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou

Visual Data in NLP

Challenge:	Multimodal Large Language Models (MLLMs) have improved document understanding performance but generate thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times.
Approach:	They propose a high-resolution document compression module to generate 324 tokens for a single document image.
Outcome:	The proposed module reduces first token latency by more than 50% and improves document comprehension performance.

Doc2Bot: Accessing Heterogeneous Documents via Conversational Bots (2022.findings-emnlp)

Copied to clipboard

Haomin Fu, Yeqin Zhang, Haiyang Yu, Jian Sun, Fei Huang, Luo Si, Yongbin Li, Cam Tu Nguyen

TACLR: A Scalable and Efficient Retrieval-based Method for Industrial Product Attribute Value Identification (2025.acl-long)

Challenge:	Documents contain various structures that hinder the ability of machines to comprehend . user information needs are often underspecified, and the nature of heterogeneous documents poses challenges.
Approach:	They propose a dataset for building machines that help users seek information via conversations . their dataset contains over 100,000 turns based on Chinese documents from five domains .
Outcome:	The proposed tasks are challenging and worthy of further research.

Copied to clipboard

Yindu Su, Huike Zou, Lin Sun, Ting Zhang, Haiyang Yang, Chen Li Yu, David Lo, Qingheng Zhang, Shuguang Han, Jufeng Chen

Green & Sustainable NLP

Challenge:	Existing methods for product attribute value identification face critical challenges . seller-provided attribute values are often incomplete or inaccurate .
Approach:	They propose a retrieval-based method that uses taxonomy-aware contrastive learning . they use product profiles and candidate values to encode and retrieve attributes based on similarity .
Outcome:	The proposed method is based on a taxonomy-aware, hard negative sampling and adaptive inference with dynamic thresholds.

Intent-Aware and Hate-Mitigating Counterspeech Generation via Dual-Discriminator Guided LLMs (2024.lrec-main)

Copied to clipboard

Haiyang Wang, Zhiliang Tian, Xin Song, Yue Zhang, Yuchen Pan, Hongkui Tu, Minlie Huang, Bin Zhou

End-to-End Beam Retrieval for Multi-Hop Question Answering (2024.naacl-long)

Challenge:	Hate speech is an aggressive expression that incites hatred towards specific groups based on their group identity.
Approach:	They propose an LLMs-based framework for counterspeech generation that uses intent-aware discriminators to decode intents of LLM models.
Outcome:	The proposed framework matches intents with hate mitigation intents and performs well.

Copied to clipboard

Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Liu Yong, Shen Huang

Information Retrieval

Challenge:	Existing beam retrieval frameworks for multi-hop question answering were customized for two-hop questions and were poorly supervised.
Approach:	They propose an end-to-end beam retrieval framework for multi-hop question answering . they combine an encoder and two classification heads to optimize the retrieval process .
Outcome:	The proposed framework improves on MuSiQue-Ans and surpasses all previous retrievers on HotpotQA and achieves 99.9% precision on 2WikiMultiHopQA.

GSID: Generative Semantic Indexing for E-Commerce Product Understanding (2025.emnlp-industry)

Copied to clipboard

Haiyang Yang, Qinye Xie, Qingheng Zhang, Chen Li Yu, Huike Zou, Chengbao Lian, Shuguang Han, Fei Huang, Jufeng Chen, Bo Zheng

Information Retrieval

Challenge:	Structured product information is a major bottleneck for the efficiency of e-commerce platforms.
Approach:	They propose a data-driven approach to generate product structured representations using product metadata.
Outcome:	Extensive experiments show that GSID can generate better product representations on real-world e-commerce platforms.

HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model (2025.acl-long)

Copied to clipboard

Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, Cheng-Lin Liu

Bridging Text and Knowledge with Multi-Prototype Embedding for Few-Shot Relational Triple Extraction (2020.coling-main)

Challenge:	Existing methods to improve instructionfollowing performance of MLLMs often trade off memory efficiency for performance gains, compromising overall efficiency.
Approach:	They propose a task-specific expansion and task-general fusion framework based on variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets.
Outcome:	The proposed framework improves performance compared to existing benchmarks.

Copied to clipboard

Haiyang Yu, Ningyu Zhang, Shumin Deng, Hongbin Ye, Wei Zhang, Huajun Chen

Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval (2024.lrec-main)

Challenge:	Existing approaches to supervised relational triple extraction require huge amounts of labeled data.
Approach:	They propose a multi-prototype embedding network model to extract the composition of relational triples from unstructured text.
Outcome:	The proposed method improves the performance of the few-shot relational triple extraction problem.

Copied to clipboard

Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model (2023.findings-emnlp)

Challenge:	Existing methods for video-text retrieval capture fine-grained semantic concepts . however, they lack the ability to capture finer-grain concepts such as objects and actions.
Approach:	They propose a dual-encoder architecture for fast video-text retrieval that learns lexicon representations to capture fine-grained semantics.
Outcome:	The proposed framework outperforms existing methods with 4.8% and 8.2% improvement on MSR-VTT and DiDeMo respectively.

Copied to clipboard

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, Fei Huang

Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement (2025.emnlp-industry)

Challenge:	Existing studies for visually-situated language understanding have shown shallow zero-shot visual text recognition ability when fed a low-resolution image with salient text information.
Approach:	They propose a model for universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM) their model is jointly finetuned on a wide range of visually situated language understanding tasks via a unified instruction format.
Outcome:	The proposed model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks across 5 domains: documents, tables, charts, natural images, and webpage screenshots.

Copied to clipboard

Haotan Guo, Jianfei He, Jiayuan Ma, Hongbin Na, Zimu Wang, Haiyang Zhang, Qi Chen, Wei Wang, Zijing Shi, Tao Shen, Ling Chen

Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)

Challenge:	Phonetic Cloaking Replacement (PCR) is a problem in content moderation in China.
Approach:	They organize PCR into a four-way surface-form taxonomy and compile PCR-ToxiCN, a dataset of 500 phonetically cloaked offensive posts gathered from the RedNote platform.
Outcome:	The proposed model achieves only an F1-score and zero-shot chain-of-thought prompting pushes performance even lower.

Copied to clipboard

Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu

DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population (2022.emnlp-demos)

Challenge:	Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling.
Approach:	They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space.
Outcome:	The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks.

Copied to clipboard

Ningyu Zhang, Xin Xu, Liankuan Tao, Haiyang Yu, Hongbin Ye, Shuofei Qiao, Xin Xie, Xiang Chen, Zhoubo Li, Lei Li

Information Extraction & Text Mining

Challenge:	Existing knowledge extraction tools are not complete due to emerging entities and relations in real-world applications.
Approach:	They propose an open-source knowledge extraction toolkit DeepKE that supports low-resource, document-level and multimodal scenarios in the knowledge base population.
Outcome:	The proposed toolkit supports low-resource, document-level and multimodal scenarios in the knowledge base population.

ExpSeek: Self-Triggered Experience Seeking for Web Agents (2026.findings-acl)

Copied to clipboard

Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, Yongbin Li

Challenge:	Existing methods for integrating experience into web agents are struggling to adapt to dynamically changing contextual observations during agent-environment interaction.
Approach:	They propose a model that shifts experience toward step-level proactive seeking by estimating step- level entropy thresholds and designing step-Level tailored experience content.
Outcome:	The proposed model achieves 9.3% and 7.5% performance improvements on Qwen3-8B and 32B models across four challenging web agent benchmarks.

TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging (2024.emnlp-main)

Copied to clipboard

Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, Fei Huang

DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling (2025.findings-acl)

Challenge:	Recent studies have shown that multimodal large language models can be useful for chart understanding, but their size limits their use in resource-constrained environments.
Approach:	They propose an efficient multimodal large language model with only 3B parameters for chart understanding.
Outcome:	The proposed model outperforms several chart-understanding MLLMs with up to 13B parameters on ChartQA, Chart-to-Text, Chart to Table, OpenCQA, and ChartX.

Copied to clipboard

Minzheng Wang, Xinghua Zhang, Kun Chen, Nan Xu, Haiyang Yu, Fei Huang, Wenji Mao, Yongbin Li

You Never Know a Person, You Only Know Their Defenses: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations (2026.findings-acl)

Challenge:	Large language models (LLMs) enabled dialogue systems are one of the central modes in human-machine interaction.
Approach:	They propose a benchmark task for dialogue element MOdeling and Element Awareness and a new benchmark for dialogue agent interaction that allows the agent to model dialogue elements via imitation learning.
Outcome:	The proposed agent performs well in both dialogue element modeling and out-of-domain tasks.

Copied to clipboard

Hongbin Na, Zimu Wang, Zhaoming Chen, Peilin Zhou, Yining Hua, Grace Ziqi Zhou, Haiyang Zhang, Tao Shen, Wei Wang, John Torous, Shaoxiong Ji, Ling Chen

ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models (2023.emnlp-demo)

Challenge:	Psychological defenses are strategies people use to manage distress.
Approach:	They propose a dialogue corpus with help seeker utterances labeled for defense level and a DMRS Co-Pilot pipeline that provides evidence-based pre-annotations.
Outcome:	The proposed framework reduces annotation time by 24.0% in a counterbalanced study.

Copied to clipboard

Chenliang Li, He Chen, Ming Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu, Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, Hongzhu Shi, Ji Zhang, Fei Huang, Jingren Zhou

Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)

Challenge:	Large language models (LLMs) have demonstrated remarkable capabilities to comprehend human intentions, engage in reasoning, and design planning-like behavior.
Approach:	They propose a framework that equips large language models with tool-use capabilities . they propose LLaMA and Chat-GLM as controllers, and a model-based agent framework .
Outcome:	The proposed framework equips open-source LLMs with tool-use capabilities . it provides a user-friendly system library with a customizable engine design .

Copied to clipboard

Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, Songfang Huang

Transforming Visual Scene Graphs to Image Captions (2023.acl-long)

Challenge:	Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model.
Approach:	They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer .
Outcome:	The proposed approach improves on the SuperGLUE and NER datasets.

Copied to clipboard

Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang

Causal Document-Grounded Dialogue Pre-training (2023.emnlp-main)

Challenge:	Existing approaches to generate captions using image captioning are based on multi-head attention (MHA)
Approach:	They propose to transform scene graphs into more descriptive captions by using multi-head attention to build a Graph Neural Network (GNN) . they construct a Mixture-of-Expert (MOE)-based decoder where each expert is built on MHA for discriminating the graph embeddings to generate different kinds of words.
Outcome:	The proposed framework can generate captions from multiple visual features and objects . it is based on a mixture-of-expert (MOE)-based decoder based upon MHA .

Copied to clipboard

Yingxiu Zhao, Bowen Yu, Bowen Li, Haiyang Yu, Jinyang Li, Chao Wang, Fei Huang, Yongbin Li, Nevin Zhang

Multi-Value-Product Retrieval-Augmented Generation for Industrial Product Attribute Value Identification (2025.emnlp-industry)

Challenge:	Existing methods for document-grounded dialogue (DocGD) rely on general pre-trained language models without a tailored pre-training approach that explicitly captures causal relationships.
Approach:	They propose a causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora and a perturbation-based strategy to capture causality.
Outcome:	The proposed strategy yields significant and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings.

Copied to clipboard

Huike Zou, Haiyang Yang, Yindu Su, Chen Li Yu, Qinye Xie, Chengbao Lian, Qingheng Zhang, Shuguang Han, Fei Huang, Jufeng Chen

Information Retrieval

Challenge:	Existing methods for product attribute value identification suffer from cascading errors and lack of generalization capability.
Approach:	They propose a multi-level retrieval scheme that uses products and attribute values as distinct hierarchical levels in PAVI domain.
Outcome:	The proposed method performs better than the state-of-the-art methods on a real-world industrial dataset.

Towards Generalized Open Information Extraction (2022.findings-emnlp)

Copied to clipboard

Bowen Yu, Zhenyu Zhang, Jingyang Li, Haiyang Yu, Tingwen Sun, Jian Liu, Yongbin Li, Bin Wang

Information Extraction & Text Mining

Challenge:	Open Information Extraction (OpenIE) models are evaluated on in-domain test sets aside from the training corpus, which violates the initial task principle of domain-independence.
Approach:	They propose to generalize OpenIE over unseen target domains with different data distributions from source training domains.
Outcome:	The proposed method beats the previous methods in both in- and out-of-domain settings by 6.0% in F1 score absolutely.

Document Set Expansion with Positive-Unlabeled Learning Using Intractable Density Estimation (2024.lrec-main)

Copied to clipboard

Haiyang Zhang, Qiuyi Chen, Yanjie Zou, Jia Wang, Yushan Pan, Mark Stevenson

Low-Resource NLP

Challenge:	Existing approaches to Document Set Expansion (DSE) rely on the unrealistic assumption of knowing the class prior for positive samples in the collection.
Approach:	They propose a novel method that utilizes intractable density estimation models to learn the class prior for positive samples in the collection.
Outcome:	The proposed method is based on a set of examples from PubMed and Covid datasets in a transductive setting.

EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models (2025.emnlp-main)

Copied to clipboard

Tao Zou, Xinghua Zhang, Haiyang Yu, Minzheng Wang, Fei Huang, Yongbin Li

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)

Challenge:	Existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect the evolution of large language models (LLMs).
Approach:	They propose to use a Segment Policy Optimization algorithm to enhance the LLM's ability to accurately fulfill multi-task workflows.
Outcome:	The proposed benchmarks show that existing benchmarks lack the complexity required to fully reflect the evolution of large language models.

Copied to clipboard

Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models (2026.acl-long)

Challenge:	Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications.
Approach:	They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) .
Outcome:	The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents.

Copied to clipboard

Lin Mu, Haiyang Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association (2025.acl-long)

Challenge:	Existing LoRA methods assume that experts operate independently, leading to unstable routing, expert dominance.
Approach:	They propose a communication-aware MoELoRA framework that relaxes this assumption by introducing expert-level communication prior to routing.
Outcome:	The proposed framework outperforms vanilla LoRA and MoELoRA on diverse language understanding tasks while maintaining expert dominance.

Copied to clipboard

Weiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag, Wenju Xu, Chen Luo, Sheikh Muhammad Sarwar, Yang Li, Hansu Gu, Hui Liu, Changlong Yu, Jiaxin Bai, Yifan Gao, Haiyang Zhang, Qi He, Shuiwang Ji, Yangqiu Song

Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs (2020.aacl-main)

Challenge:	Goal-oriented script planning is used by humans to plan for typical activities . however, this capability remains underexplored due to several challenges .
Approach:	They propose a framework that enables product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions.
Outcome:	The proposed framework can generate product-enriched scripts from 2.4 million scripts . human annotations are conducted to provide gold labels for a sampled subset .

Copied to clipboard

Haiyang Zhang, Alison Sneyd, Mark Stevenson

FAQ: Mitigating Quantization Error via Regenerating Calibration Data with Family-Aware Quantization (2026.findings-acl)

Challenge:	Existing methods to quantify gender bias in word embeddings are not robust and cannot identify common types of bias.
Approach:	They propose to quantify gender bias by using cosine similarity to a pair of gender words and using analogies.
Outcome:	The proposed methods are not robust and cannot identify common types of bias, while analogies are unsuitable indicators.

Copied to clipboard

Haiyang Xiao, Weiqing Li, Jinyue Guo, Guochao Jiang, Guohua Liu, Yuewei Zhang

Challenge:	representativeness and universality of calibration data remain a bottleneck in quantization accuracy.
Approach:	They propose a framework that leverages prior knowledge from LLMs to generate calibration samples . their framework reduces accuracy loss by up to 28.5% compared to baseline .
Outcome:	Experiments show that family-aware quantization reduces accuracy loss by up to 28.5% compared to baseline data.

AEGIS: A Holistic Benchmark for Evaluating Forensic Analysis of AI-Generated Academic Images (2026.acl-long)

Copied to clipboard

Bo Zhang, Tzu-Yen Ma, Zichen Tang, Junpeng Ding, Zirui Wang, Yizhuo Zhao, Peilin Gao, Zijie Xi, Zixin Ding, Haiyang Sun, Haocheng Gao, Yuan Liu, Liangjia Wang, Yiling Huang, Yujie Wang, Yuyue Zhang, Ronghui Xi, Yuanze Li, Jiacheng Liu, Zhongjun Yang, Haihong E

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022.emnlp-main)

Challenge:	AEGIS examines whether current models can effectively audit AI-generated images in academic papers.
Approach:	They propose a holistic benchmark for forensic analysis of AI-Generated academic ImageS that reveals limitations in academic image forensics.
Outcome:	AEGIS compared with existing benchmarks on seven academic categories and features key advances in forensic analysis.

Copied to clipboard

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

Visual Data in NLP

Challenge:	Existing pre-trained vision-language models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment.
Approach:	They propose a vision-language foundation model with cross-modal skip-connections that can be pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives.
Outcome:	The proposed model achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering.

EvoRoute: Experience-Driven Self-Routing LLM Agent Systems (2026.acl-long)

Copied to clipboard

Guibin Zhang, Haiyang Yu, Kaiming Yang, Bingli Wu, Fei Huang, Yongbin Li, Shuicheng Yan

ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)

Challenge:	EvoRoute is a self-evolving model routing paradigm that transcends static, pre-defined model assignments.
Approach:	They propose a model routing paradigm that transcends static, pre-defined model assignments.
Outcome:	Experiments on GAIA and BrowseComp+ show that EvoRoute reduces execution cost and latency by over 70%.

Copied to clipboard

Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, Cheng-Lin Liu

OpenUE: An Open Toolkit of Universal Extraction from Text (2020.emnlp-demos)

Challenge:	Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency.
Approach:	They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task.
Outcome:	The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation.

Copied to clipboard

Ningyu Zhang, Shumin Deng, Zhen Bi, Haiyang Yu, Jiacheng Yang, Mosha Chen, Fei Huang, Wei Zhang, Huajun Chen

Information Extraction & Text Mining

Challenge:	a large number of natural language processing tasks focus on token-level or sentence-level understandings.
Approach:	They propose an open-source and extensible toolkit for various extraction tasks . they deploy an online demo with restful APIs to support real-time extraction .
Outcome:	The proposed model can be used to extract information from text without training and deployment.

MemPO: Self-Memory Policy Optimization for Long-Horizon Agents (2026.findings-acl)

Copied to clipboard

Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo, Yongbin Li, Jinli Suo

Challenge:	Existing methods for long-horizon agents introduce the external memory module and look up the relevant information from the stored memory, which prevents the model from proactively managing its memory content and aligning with the agent’s overarching task objectives.
Approach:	They propose an algorithm which enables agents to autonomously manage their memory during interaction with environment and selectively retain crucial information.
Outcome:	Extensive experiments show that the proposed algorithm achieves absolute F1 score gains of 25.98 over the base model and 7.1 over the previous SOTA baseline while preserving task performance.

TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection (2022.emnlp-main)

Copied to clipboard

Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, Songfang Huang

Visual Data in NLP

Challenge:	Existing vision-and-language pre-training models suffer from long visual sequences . experimental results show that TRIPS gains a speedup of 40% over previous similar VLP models .
Approach:	They propose an efficient vision-and-language pre-training model with text-relevant image patch selection, TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference.
Outcome:	The proposed model can speed up training and inference by 40% over previous models.

Beyond Sentence-level Labels: Integrating Conversational Context and Personal Experience for Natural Emotional Expression (2026.findings-acl)

Copied to clipboard

Haiyang Sun, Chenyang Le, Wei Wang, Leying Zhang, Chuang Li, Bing Han, Chenda Li, Mengxiao Bi, Yanmin Qian