Papers by Shih-Fu Chang
Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining. |
| Approach: | They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts. |
| Outcome: | The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings . |
Cross-media Structured Common Space for Multimedia Event Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | We propose a new task to extract events and their arguments from multimedia documents . traditional methods target text, images or videos, but multimedia content is distributed via multimedia . |
| Approach: | They propose a method that encodes structured representations of semantic information from textual and visual data into a common embedding space. |
| Outcome: | The proposed method achieves 4.0% and 9.8% absolute gains on text event argument role labeling and visual event extraction. |
Cross-lingual Structure Transfer for Relation and Event Extraction (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to identify complex semantic structures are difficult to train from under-annotated sources. |
| Approach: | They exploit relation- and event-relevant language-universal features to train relation or event extractors from source annotations and apply them to target languages. |
| Outcome: | The proposed approach achieves comparable performance to state-of-the-art models trained on 3,000 manually annotated mentions. |
Joint Multimedia Event Extraction from Video and Article (2021.findings-emnlp)
Copied to clipboard
Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, Shih-Fu Chang
| Challenge: | Existing methods to extract multimedia events from video and text are limited to video and images. |
| Approach: | They propose a task to jointly extract events from video and text documents . they propose 'self-supervised' cross-modal event coreference model and cross-mod transformer architecture . |
| Outcome: | The proposed method achieves 6.0% and 5.8% absolute F-score gain on video-article pairs . the proposed method can resolve coreference and extract multimodal event frames more accurately than existing methods. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models (2023.findings-emnlp)
Copied to clipboard
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing approaches to decompose VL reasoning rely on domain-specific sub-question decomposing models. |
| Approach: | They propose a framework that iteratively decomposes VL reasoning using large language models. |
| Outcome: | The proposed framework outperforms existing models on multiple VL reasoning tasks. |
Weakly-Supervised Temporal Article Grounding (2022.emnlp-main)
Copied to clipboard
Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher Thomas, Hammad Ayyubi, Heng Ji, Shih-Fu Chang
| Challenge: | Existing VG models make unrealistic assumptions about how to ground video segments . a recent study has shown that video grounding can be useful for downstream applications . |
| Approach: | They propose a new task: Weakly-Supervised temporal Article Grounding (WSAG) given an article and a relevant video, WSAG aims to localize all "groundable" sentences to the video. |
| Outcome: | The proposed method is simple but effective, and it can be used in real-world applications. |
Cross-lingual Structure Transfer for Zero-resource Event Extraction (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches for information extraction only use name tagging . Currently, most successful cross-lingual transfer learning methods are limited to sequence labeling . |
| Approach: | They propose a share-and-transfer framework to transfer graph structures across languages . they propose to convert sentences in any language to language-universal graph structures . |
| Outcome: | The proposed framework performs comparable to state-of-the-art models on three languages without annotations. |
M2-TabFact: Multi-Document Multi-Modal Fact Verification with Visual and Textual Representations of Tabular Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing fact-checking systems that can reason over structured data are inefficient compared to humans. |
| Approach: | They propose a multi-modal table-based fact verification task that requires reasoning over visual and textual representations of structured data. |
| Outcome: | The proposed model can reason over visual and textual representations of structured data. |
RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System (2021.naacl-demos)
Copied to clipboard
Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Heng Ji
| Challenge: | We present a new information extraction system that can construct temporal event graphs from news documents. |
| Approach: | They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction . |
| Outcome: | The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities. |
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)
Copied to clipboard
Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation (2021.naacl-demos)
Copied to clipboard
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Ranran Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Yi Fung, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, Jasmine Rah, David Liem, Ahmed ELsayed, Martha Palmer, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
| Challenge: | a new framework to digest relevant biomedical knowledge is needed to combat COVID-19 . quantity of research results is a bottleneck, and false information promoted in publications . |
| Approach: | a team of researchers has developed a framework to extract multimedia knowledge elements from scientific literature to combat COVID-19. |
| Outcome: | a new framework extracts fine-grained multimedia knowledge elements from scientific literature . it provides detailed contextual sentences, subfigures, and knowledge subgraphs as evidence . the framework is based on a case study of drug repurposing . |
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Visual scenes often involve multiple people and humans can distinguish between them based on context descriptions about what happened before, their mental/physical states, and intentions. |
| Approach: | They propose a task that tests human-centric commonsense grounding models' ability to distinguish individuals given context descriptions about what happened before and their mental/physical states or intentions. |
| Outcome: | The proposed model outperforms pre-trained and non-pretrained models on 130k commonsense descriptions annotated on 67k images. |
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models require large amounts of image-caption data for pre-training . existing models require expensive data collection and curation . |
| Approach: | They propose to conduct "mask-and-predict" pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. |
| Outcome: | The proposed approach achieves performance close to a model pre-trained with aligned data, on four English benchmarks. |
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection (2021.acl-long)
Copied to clipboard
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, Avi Sil
| Challenge: | a novel approach to detect fake news is needed due to training data scarcity . current methods focus on document-level fake news detection using lexical features and semantic embeddings . |
| Approach: | They propose a novel benchmark for fake news detection at the knowledge element level . they propose synthesis method which manipulates knowledge elements to generate noisy training data . |
| Outcome: | The proposed method outperforms the state-of-the-art in detecting misinformation . it yields fine-grained explanations and outperformed the current methods . |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
GAIA: A Fine-grained Multimedia Knowledge Extraction System (2020.acl-demos)
Copied to clipboard
Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, Daniel Napierski, Marjorie Freedman
| Challenge: | Open source knowledge extraction tools are used for many real-world applications, but there is no comprehensive system for KE. |
| Approach: | They propose a multimedia knowledge extraction system that takes multimedia data from various sources and languages as input and creates a coherent, structured knowledge base. |
| Outcome: | The system achieves top performance at the recent NIST TAC SM-KBP2019 evaluation. |
UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | supervised methods for vision-language tasks have been well-studied, but they lack the fine-grained information needed for semantics understanding. |
| Approach: | They propose a framework to take advantage of fine-grained information for zero-shot vision-language learning, covering multiple tasks such as VQA, SNLI-VE, and VCR. |
| Outcome: | The proposed framework outperforms previous zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR. |
Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that understand image and text but also cross-reference in-between are lacking in evaluation data resources. |
| Approach: | They propose a multimodal evaluation pipeline to automatically generate question-answer pairs to test models’ understanding of the visual scene, text, and related knowledge. |
| Outcome: | The proposed model can answer the highly semantic VCR question correctly but fails to answer related visual question (Q2), textual question (q3), and background knowledge question ( Q4) as shallow mappings with language priors and unbalanced utilization of information between modalities. |
Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond (2023.findings-emnlp)
Copied to clipboard
Zhecan Wang, Long Chen, Haoxuan You, Keyang Xu, Yicheng He, Wenhao Li, Noel Codella, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing studies have examined dataset biases in VQA benchmarks with short-phrase answers Multiple-choice Question with the LONG Answers (VCR, VLEP, etc.) |
| Approach: | They propose to use Adversarial Data Synthesis (ADS) to generate synthetic training and debiased evaluation data and introduce Intra-sample Counterfactual Training (ICT) to assist models in utilizing synthesized training data. |
| Outcome: | The proposed approach improves model performance even in domain-shifted scenarios. |
PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction (2025.findings-naacl)
Copied to clipboard
| Challenge: | PuzzleGPT is a modular and iterative puzzlesolving method for predicting time and location from images. |
| Approach: | They propose to formalize this ability into core skills and implement it using different modules in an expert pipeline called PuzzleGPT. |
| Outcome: | The proposed method outperforms large VLMs and finetuned models on TARA and WikiTilo and rivals or surpasses finetuned models. |
Non-Sequential Graph Script Induction via Multimedia Grounding (2023.acl-long)
Copied to clipboard
| Challenge: | Existing scripts for everyday tasks are presented in a linear manner, which does not reflect the flexibility displayed by people executing tasks in real life. |
| Approach: | They propose to use loosely aligned videos to train a non-sequential graph script induction task by using a multimodal framework to ground procedural videos to WikiHow textual steps. |
| Outcome: | The proposed model outperforms the WikiHow linear baseline by 48.76% . it can predict future steps given a partial step sequence and generate explicit graph scripts . |
Personalized Video Comment Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Generating personalized responses in video poses a unique challenge for language models. |
| Approach: | They propose a new automatic metric based on Large Language Models with few-shot in-context learning that measures quality from the aspects of emotion, language style and content relevance. |
| Outcome: | The proposed metric measures quality from emotion, language style and content relevance with human evaluations. |
Incorporating Background Knowledge into Video Description Generation (D18-1)
Copied to clipboard
| Challenge: | Existing methods for video captioning focus on generating generic descriptions that lack contextual knowledge. |
| Approach: | They propose a method that uses video meta-data to retrieve topically related news documents for a video and extracts the events and named entities from these documents. |
| Outcome: | The proposed model is based on a news video dataset and is evaluated on it. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning (2024.findings-acl)
Copied to clipboard
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji
| Challenge: | LVLMs are known for producing text that is factually inconsistent with visual input . factuality of generated captions for structured visuals has not been studied as much . |
| Approach: | They propose a typology of factual errors in captions generated by large vision-language models . they propose CHOCOLATE, a visual entailment model that outperforms current models based on this analysis . |
| Outcome: | The proposed model outperforms current models in evaluating caption factuality. |
Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to understand chart plots are difficult to apply to visual-language tasks. |
| Approach: | They propose a V+L model that learns how to interpret table information from chart images via cross-modal pre-training on plot table pairs. |
| Outcome: | The proposed model outperforms state-of-the-art models on the chartQA benchmark by over 8% performance gains. |