Papers by Heng Chang
Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining. |
| Approach: | They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts. |
| Outcome: | The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings . |
Cross-media Structured Common Space for Multimedia Event Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | We propose a new task to extract events and their arguments from multimedia documents . traditional methods target text, images or videos, but multimedia content is distributed via multimedia . |
| Approach: | They propose a method that encodes structured representations of semantic information from textual and visual data into a common embedding space. |
| Outcome: | The proposed method achieves 4.0% and 9.8% absolute gains on text event argument role labeling and visual event extraction. |
Cross-lingual Structure Transfer for Relation and Event Extraction (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to identify complex semantic structures are difficult to train from under-annotated sources. |
| Approach: | They exploit relation- and event-relevant language-universal features to train relation or event extractors from source annotations and apply them to target languages. |
| Outcome: | The proposed approach achieves comparable performance to state-of-the-art models trained on 3,000 manually annotated mentions. |
Joint Multimedia Event Extraction from Video and Article (2021.findings-emnlp)
Copied to clipboard
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, Shih-Fu Chang
| Challenge: | Existing methods to extract multimedia events from video and text are limited to video and images. |
| Approach: | They propose a task to jointly extract events from video and text documents . they propose 'self-supervised' cross-modal event coreference model and cross-mod transformer architecture . |
| Outcome: | The proposed method achieves 6.0% and 5.8% absolute F-score gain on video-article pairs . the proposed method can resolve coreference and extract multimodal event frames more accurately than existing methods. |
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization has shown promise for Large Language Models, but current methods require lengthy training to alleviate quantization loss. |
| Approach: | They propose to decouple weights and incorporate Low-Rank adapters to reduce weight sharing . they validate the approach on LLaMA2 families and Mistral on downstream evaluation . |
| Outcome: | The proposed approach shows high performance while reducing deployment time faced with multiple scenarios. |
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks (2026.acl-long)
Copied to clipboard
Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, Heng Ji
| Challenge: | Existing research exposes multimodal large language models to knowledge poisoning attacks . localized poisoning attack achieves up to 56% success rate even under restricted access . globalized poison attack completely disrupts model generation to 0% accuracy with just one poisoned content. |
| Approach: | They propose a framework to study the vulnerability of multimodal RAG under knowledge poisoning attacks. |
| Outcome: | The proposed framework exploits two new attack strategies on multimodal RAGs under knowledge poisoning. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
Weakly-Supervised Temporal Article Grounding (2022.emnlp-main)
Copied to clipboard
Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang
| Challenge: | Existing VG models make unrealistic assumptions about how to ground video segments . a recent study has shown that video grounding can be useful for downstream applications . |
| Approach: | They propose a new task: Weakly-Supervised temporal Article Grounding (WSAG) given an article and a relevant video, WSAG aims to localize all "groundable" sentences to the video. |
| Outcome: | The proposed method is simple but effective, and it can be used in real-world applications. |
SYNTHIA: Novel Concept Design with Affordance Composition (2025.acl-long)
Copied to clipboard
Hyeonjeong Ha, Xiaomeng Jin, Jeonghwan Kim, Jiateng Liu, Zhenhailong Wang, Khanh Duy Nguyen, Ansel Blume, Nanyun Peng, Kai-Wei Chang, Heng Ji
| Challenge: | Existing studies on concept design using text-to-image models have enabled rapid ideation of novel visual concepts. |
| Approach: | They propose a framework for generating novel, functionally coherent designs based on desired affordances by decomposing concepts into parts and affordance . they also develop a curriculum learning scheme that fine-tunes T2I models to progressively learn affordance composition while maintaining visual novelty. |
| Outcome: | The proposed framework outperforms state-of-the-art models for novelty and functional coherence in human evaluation. |
From Implicit Graph Encoding to Explicit Evidence: A Training-Free LLM Framework for Temporal Knowledge Graph Reasoning (2026.findings-acl)
Copied to clipboard
Guo Tang, Ke Cheng, Huiming Fan, Heng Chang, Wenxiang Zheng, Xianhao Ou, Junjia Xiang, Ming Liu, Yujun Zhou, Li Lanyu, Bing Qin
| Challenge: | Existing Large Language Models (LLMs) struggle with implicit modality alignment and suboptimal graph linearization. |
| Approach: | They propose a training-free, test-time adaptive framework that reframes TKG prediction as explicit evidence-driven reasoning. |
| Outcome: | ExE-LLM outperforms fully trained graph neural networks on four benchmarks . it achieves SOTA performance in inductive settings, significantly outperforming fully trained neural networks . |
Cross-lingual Structure Transfer for Zero-resource Event Extraction (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches for information extraction only use name tagging . Currently, most successful cross-lingual transfer learning methods are limited to sequence labeling . |
| Approach: | They propose a share-and-transfer framework to transfer graph structures across languages . they propose to convert sentences in any language to language-universal graph structures . |
| Outcome: | The proposed framework performs comparable to state-of-the-art models on three languages without annotations. |
RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System (2021.naacl-demos)
Copied to clipboard
Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Heng Ji
| Challenge: | We present a new information extraction system that can construct temporal event graphs from news documents. |
| Approach: | They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction . |
| Outcome: | The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities. |
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)
Copied to clipboard
Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities (2025.findings-acl)
Copied to clipboard
Shixin Jiang, Jiafeng Liang, Jiyuan Wang, Xuan Dong, Heng Chang, Weijiang Yu, Jinhua Du, Ming Liu, Bing Qin
| Challenge: | MLLMs are able to integrate multiple modalities into a single model to tackle complex tasks in real-world scenarios. |
| Approach: | They propose a comprehensive survey of Omni-MLLMs to address the challenges and opportunities of multimodal modeling. |
| Outcome: | The proposed model can integrate multiple modalities into a single model and provide novel perspectives. |
MACAROON: Training Vision-Language Models To Be Your Engaged Partners (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) generate detailed responses even when questions are ambiguous or unanswerable, leading to hallucinations and bias issues. |
| Approach: | They propose a three-tiered hierarchy for questions of invalid, ambiguous, and personalizable nature to measure the proactive engagement capabilities of LVLMs. |
| Outcome: | The proposed model generates contrastive response pairs for unlabeled questions, achieving 0.84 AAR, while maintaining comparable performance on general tasks. |
COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation (2021.naacl-demos)
Copied to clipboard
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Ranran Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Yi Fung, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, Jasmine Rah, David Liem, Ahmed ELsayed, Martha Palmer, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
| Challenge: | a new framework to digest relevant biomedical knowledge is needed to combat COVID-19 . quantity of research results is a bottleneck, and false information promoted in publications . |
| Approach: | a team of researchers has developed a framework to extract multimedia knowledge elements from scientific literature to combat COVID-19. |
| Outcome: | a new framework extracts fine-grained multimedia knowledge elements from scientific literature . it provides detailed contextual sentences, subfigures, and knowledge subgraphs as evidence . the framework is based on a case study of drug repurposing . |
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism (2025.acl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have shown exceptional performance in multimodal tasks, but their effectiveness in complex visual reasoning is constrained. |
| Approach: | They propose a training-free approach that enhances Reasoning in Large Vision-Language Models . they propose integrating Monte Carlo Tree Search and Self-Reward mechanisms into the reasoning tree . |
| Outcome: | The proposed approach surpasses current prompting methods and secures state-of-the-art performance across three multimodal reasoning benchmarks. |
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection (2021.acl-long)
Copied to clipboard
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, Avi Sil
| Challenge: | a novel approach to detect fake news is needed due to training data scarcity . current methods focus on document-level fake news detection using lexical features and semantic embeddings . |
| Approach: | They propose a novel benchmark for fake news detection at the knowledge element level . they propose synthesis method which manipulates knowledge elements to generate noisy training data . |
| Outcome: | The proposed method outperforms the state-of-the-art in detecting misinformation . it yields fine-grained explanations and outperformed the current methods . |
How do Language Models Reshape Entity Alignment? A Survey of LM-Driven EA Methods: Advances, Benchmarks, and Future (2025.emnlp-main)
Copied to clipboard
| Challenge: | Entity alignment (EA) is critical for knowledge graph (KG) integration. |
| Approach: | They propose a taxonomy that categorizes methods in three stages: data preparation, feature embedding, and alignment. |
| Outcome: | The proposed taxonomy categorizes methods in three key stages: data preparation, feature embedding, and alignment. |
TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2024.findings-acl)
Copied to clipboard
Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji
| Challenge: | Recent studies suggest that event extraction evaluations may not accurately reflect the true performance. |
| Approach: | They propose a standardized, fair, and reproducible benchmark for event extraction . they use standardized scripts and splits for 16 datasets spanning eight domains . |
| Outcome: | The proposed benchmarks show that they struggle to achieve satisfactory performance. |
Reusing Transferable Weight Increments for Low-resource Style Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Text style transfer (TST) is crucial in natural language processing, aiming to endow text with a new style without altering its meaning. |
| Approach: | They propose a framework to use style features in weight increments to transfer low-resource styles effectively. |
| Outcome: | The proposed framework achieves remarkable performance across different backbones, achieving particularly effective results in low-resource scenarios. |
GAIA: A Fine-grained Multimedia Knowledge Extraction System (2020.acl-demos)
Copied to clipboard
Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, Daniel Napierski, Marjorie Freedman
| Challenge: | Open source knowledge extraction tools are used for many real-world applications, but there is no comprehensive system for KE. |
| Approach: | They propose a multimedia knowledge extraction system that takes multimedia data from various sources and languages as input and creates a coherent, structured knowledge base. |
| Outcome: | The system achieves top performance at the recent NIST TAC SM-KBP2019 evaluation. |
A Context-Dependent Gated Module for Incorporating Symbolic Semantics into Event Coreference Resolution (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for event coreference resolution use symbolic features, but they are noisy and contain errors. |
| Approach: | They propose a context-dependent gated module to adaptively control the information flows from the input symbolic features. |
| Outcome: | The proposed model achieves state-of-the-art on two datasets: ACE 2005 and KBP 2016 . |
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition (D18-1)
Copied to clipboard
| Challenge: | a novel information network decipherment paradigm is proposed for fine-grained coordinated cross-lingual text stream alignment. |
| Approach: | They propose to use Burst Information Networks as media to represent text streams . they propose a simple yet effective information network decipherment algorithm with diverse clues . |
| Outcome: | The proposed approach outperforms existing approaches on bilingual lexicon extraction from coordinated text streams and can harvest high-quality alignments from large amounts of streaming data for endless language knowledge mining. |
Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances with LLMs have shown promising results across various tasks, but their use in answering questions from knowledge bases remains largely unexplored. |
| Approach: | They propose a framework that utilizes an LLM-based agent with multiple roles for KBQA tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks yielding F1 scores of 11.8% and 20.7%, respectively. |
Non-Sequential Graph Script Induction via Multimedia Grounding (2023.acl-long)
Copied to clipboard
| Challenge: | Existing scripts for everyday tasks are presented in a linear manner, which does not reflect the flexibility displayed by people executing tasks in real life. |
| Approach: | They propose to use loosely aligned videos to train a non-sequential graph script induction task by using a multimodal framework to ground procedural videos to WikiHow textual steps. |
| Outcome: | The proposed model outperforms the WikiHow linear baseline by 48.76% . it can predict future steps given a partial step sequence and generate explicit graph scripts . |
Incorporating Background Knowledge into Video Description Generation (D18-1)
Copied to clipboard
| Challenge: | Existing methods for video captioning focus on generating generic descriptions that lack contextual knowledge. |
| Approach: | They propose a method that uses video meta-data to retrieve topically related news documents for a video and extracts the events and named entities from these documents. |
| Outcome: | The proposed model is based on a news video dataset and is evaluated on it. |
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning (2024.findings-acl)
Copied to clipboard
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji
| Challenge: | LVLMs are known for producing text that is factually inconsistent with visual input . factuality of generated captions for structured visuals has not been studied as much . |
| Approach: | They propose a typology of factual errors in captions generated by large vision-language models . they propose CHOCOLATE, a visual entailment model that outperforms current models based on this analysis . |
| Outcome: | The proposed model outperforms current models in evaluating caption factuality. |
Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to understand chart plots are difficult to apply to visual-language tasks. |
| Approach: | They propose a V+L model that learns how to interpret table information from chart images via cross-modal pre-training on plot table pairs. |
| Outcome: | The proposed model outperforms state-of-the-art models on the chartQA benchmark by over 8% performance gains. |