Papers by Heng Lu

Cross-media Structured Common Space for Multimedia Event Extraction (2020.acl-main)

Copied to clipboard

Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang

Challenge:	We propose a new task to extract events and their arguments from multimedia documents . traditional methods target text, images or videos, but multimedia content is distributed via multimedia .
Approach:	They propose a method that encodes structured representations of semantic information from textual and visual data into a common embedding space.
Outcome:	The proposed method achieves 4.0% and 9.8% absolute gains on text event argument role labeling and visual event extraction.

Cross-lingual Structure Transfer for Relation and Event Extraction (D19-1)

Copied to clipboard

Ananya Subburathinam, Di Lu, Heng Ji, Jonathan May, Shih-Fu Chang, Avirup Sil, Clare Voss

Multilinguality

Challenge:	Existing approaches to identify complex semantic structures are difficult to train from under-annotated sources.
Approach:	They exploit relation- and event-relevant language-universal features to train relation or event extractors from source annotations and apply them to target languages.
Outcome:	The proposed approach achieves comparable performance to state-of-the-art models trained on 3,000 manually annotated mentions.

Platforms for Non-speakers Annotating Names in Any Language (P18-4)

Copied to clipboard

Ying Lin, Cash Costello, Boliang Zhang, Di Lu, Heng Ji, James Mayfield, Paul McNamee

Challenge:	Traditionally, native speakers of a language have been asked to annotate a corpus in that language.
Approach:	They propose two annotation platforms that allow an English speaker to annotate names for any language without knowing the language.
Outcome:	The proposed annotations achieved state-of-the-art performance on two surprise languages and ten languages at TAC-KBP EDL2017.

Entity-aware Image Caption Generation (D18-1)

Copied to clipboard

Di Lu, Spencer Whitehead, Lifu Huang, Heng Ji, Shih-Fu Chang

Multimodality

Challenge:	Existing image captioning approaches generate generic descriptions of visual content and ignore background information.
Approach:	They propose a task which generates informative image captions using images and hashtags as input.
Outcome:	The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr.

Efficient Attentions for Long Document Summarization (2021.naacl-main)

Copied to clipboard

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, Lu Wang

Responsible & Trustworthy NLP

Challenge:	Existing models that use full attentions have quadratic computational and memory complexities, and are too costly for long documents.
Approach:	They propose an efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source.
Outcome:	The proposed model can process ten times more tokens than current models that use full attentions.

ELISA-EDL: A Cross-lingual Entity Extraction, Linking and Localization System (N18-5)

Copied to clipboard

Boliang Zhang, Ying Lin, Xiaoman Pan, Di Lu, Jonathan May, Kevin Knight, Heng Ji

Information Extraction & Text Mining

Challenge:	ELISA-EDL is a cross-lingual entity extraction, linking and localization system for Wikipedia languages.
Approach:	They propose a cross-lingual entity extraction, linking and localization system for English speakers . it extracts entities from unstructured text in any of 282 Wikipedia languages and links them to English knowledge bases .
Outcome:	The proposed system extracts entity mentions from Wikipedia and links them to English knowledge bases and visualizes locations related to disaster topics on a world heatmap.

BadWindtunnel: Defending Backdoor in High-noise Simulated Training with Confidence Variance (2025.findings-acl)

Copied to clipboard

Ruyi Zhang, Songlei Jian, Yusong Tan, Heng Gao, Haifang Zhou, Kai Lu

Robustness in NLP

Challenge:	Current backdoor attack defenders in NLP typically involve data reduction or model pruning, risking losing crucial information.
Approach:	They propose a backdoor defender that allows precise control over training conditions to model backdoor learning behavior without affecting the final model.
Outcome:	The proposed model reduces the backdoor learning behavior without affecting the final model.

Cross-lingual Structure Transfer for Zero-resource Event Extraction (2020.lrec-1)

Copied to clipboard

Di Lu, Ananya Subburathinam, Heng Ji, Jonathan May, Shih-Fu Chang, Avi Sil, Clare Voss

Information Extraction & Text Mining

Challenge:	Existing approaches for information extraction only use name tagging . Currently, most successful cross-lingual transfer learning methods are limited to sequence labeling .
Approach:	They propose a share-and-transfer framework to transfer graph structures across languages . they propose to convert sentences in any language to language-universal graph structures .
Outcome:	The proposed framework performs comparable to state-of-the-art models on three languages without annotations.

Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)

Copied to clipboard

Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, Wai Lam

Language Models

Challenge:	Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood.
Approach:	They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs.
Outcome:	The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks.

FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio Generation (2025.acl-long)

Copied to clipboard

Huadai Liu, Jialei Wang, Rongjie Huang, Yang Liu, Heng Lu, Zhou Zhao, Wei Xue

Multimodality

Challenge:	Recent advances in latent diffusion models (LDMs) have markedly enhanced text-to-audio generation, yet their iterative sampling processes impose substantial computational demands, limiting practical deployment.
Approach:	They propose to learn straight flow for fast simulation by using flashAudio with rectified flows and immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment.
Outcome:	The proposed method can learn straight flow for fast simulations and reduce noise distribution.

Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning (2026.findings-acl)

Copied to clipboard

Ganghao Liu, Qin Zhou, Zhe Wang, Xuehan Lu, Haihua Huang, Yunfei Tong, Heng Tian

Language Models

Challenge:	Parameter-efficient fine-tuning (PEFT) is a low-cost alternative to full fine-timing due to the massive overhead.
Approach:	They propose a Mixture-of-Experts approach that enhances specialization while maintaining low resource overhead.
Outcome:	The proposed approach outperforms or matches state-of-the-art methods on GLUE, GSM8K, MBPP, and a text rewriting task from SmolTalk.

CA-GAR: Context-Aware Alignment of LLM Generation for Document Retrieval (2025.findings-acl)

Copied to clipboard

Heng Yu, Junfeng Kang, Rui Li, Qi Liu, Liyang He, Zhenya Huang, Shuanghong Shen, Junyu Lu

Information Retrieval

Challenge:	Recent techniques such as Generation-Augmented Retrieval (GAR) and Generative Document Retrieleval (GDR) leverage LLMs to enhance retrieval performance but face key challenges: GAR’s generated content may not always align with the target document corpus, while GDR limits the generative capacity of LLM.
Approach:	They propose a Context-Aware Generation-Augmented Retrieval approach which integrates corpus information into their generation process.
Outcome:	Experimental results show that CA-GAR outperforms existing methods on seven tasks and four non-English languages.

UniSpeaker: A Unified Approach for Multimodality-driven Speaker Generation (2025.findings-emnlp)

Copied to clipboard

Zhengyan Sheng, Zhihao Du, Heng Lu, ShiLiang Zhang, Zhen-Hua Ling

Multimodality

Challenge:	a new framework for speaker generation is proposed to enable multimodal speaker generation . multimodal cues such as visual appearance, textual descriptions, and other biometric signals are still in its early stages.
Approach:	a new framework is proposed to enable multimodal speaker generation . the framework uses self-distillation to apply speaker disentanglement to speech generation a model is developed .
Outcome:	The proposed framework is the first to support unified voice generation from arbitrary modality combinations.

Error Analysis of Uyghur Name Tagging: Language-specific Techniques and Remaining Challenges (L18-1)

Copied to clipboard

Halidanmu Abudukelimu, Abudoukelimu Abulizi, Boliang Zhang, Xiaoman Pan, Di Lu, Heng Ji, Yang Liu

Syntactic Text Processing

Challenge:	despite efforts at name tagging, there is limited understanding on the performance ceiling . despite the high-resource language, there are very few natural language processing tools available .
Approach:	They propose to use a machine learning model to identify Uyghur name tagger errors . they conclude that such a model is unlikely to be effective for Uygur, or low-resource languages .
Outcome:	The proposed model is unlikely to be effective for Uyghur, or low-resource languages in general, the authors argue . they show that the proposed model can be used for high-res languages with superficial features .

Symbolization, Prompt, and Classification: A Framework for Implicit Speaker Identification in Novels (2023.findings-emnlp)

Copied to clipboard

Yue Chen, Tianwei He, Hongbin Zhou, Jia-Chen Gu, Heng Lu, Zhen-Hua Ling

Low-Resource NLP

Challenge:	Existing methods for speaker identification in novel dialogues are limited to handling explicit narrative patterns and complex cases.
Approach:	They propose a framework which identifies implicit speakers in novels via symbolization, prompt, and classification.
Outcome:	The proposed framework outperforms existing methods by 4.8% accuracy on the web novel collection, which reduces 47% of speaker identification errors, and outperfies the emerging ChatGPT.

Visual Attention Model for Name Tagging in Multimodal Social Media (P18-1)

Copied to clipboard

Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, Heng Ji

Multimodality

Challenge:	Name tagging is a key task for language understanding, but is often limited by the short textual components.
Approach:	They propose a novel model architecture based on visual attention that outperforms other methods . they use multimodal datasets to analyze the name tagging task on social media .
Outcome:	The proposed model outperforms existing methods and significantly outperformed existing methods.

Closing the Modality Reasoning Gap for Speech Large Language Models (2026.acl-long)

Copied to clipboard

Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu

Multimodality

Challenge:	Recent advances in Speech Large Language Models have a modality reasoning gap that is not addressed by prior work.
Approach:	They propose a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design.
Outcome:	Experiments on MMSU and OBQA show that the proposed framework narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.

Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering (2024.emnlp-main)

Copied to clipboard

Chang Zong, Yuchen Yan, Weiming Lu, Jian Shao, Yongfeng Huang, Heng Chang, Yueting Zhuang

Natural Language Interfaces

Challenge:	Recent advances with LLMs have shown promising results across various tasks, but their use in answering questions from knowledge bases remains largely unexplored.
Approach:	They propose a framework that utilizes an LLM-based agent with multiple roles for KBQA tasks.
Outcome:	The proposed framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks yielding F1 scores of 11.8% and 20.7%, respectively.