Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, Shih-Fu Chang
| Challenge: | Existing methods to extract multimedia events from video and text are limited to video and images. |
| Approach: | They propose a task to jointly extract events from video and text documents . they propose 'self-supervised' cross-modal event coreference model and cross-mod transformer architecture . |
| Outcome: | The proposed method achieves 6.0% and 5.8% absolute F-score gain on video-article pairs . the proposed method can resolve coreference and extract multimodal event frames more accurately than existing methods. |
Similar Papers
Cross-media Structured Common Space for Multimedia Event Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | We propose a new task to extract events and their arguments from multimedia documents . traditional methods target text, images or videos, but multimedia content is distributed via multimedia . |
| Approach: | They propose a method that encodes structured representations of semantic information from textual and visual data into a common embedding space. |
| Outcome: | The proposed method achieves 4.0% and 9.8% absolute gains on text event argument role labeling and visual event extraction. |
Three Stream Based Multi-level Event Contrastive Learning for Text-Video Event Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event extraction ignore motion representations in videos and are misguided by background noise. |
| Approach: | They propose a text-video based multimodal event extraction framework that integrates video appearance features and motion representations with video appearance. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods in the event extraction field. |
Harvesting Events from Multiple Sources: Towards a Cross-Document Event Extraction Paradigm (2024.findings-acl)
Copied to clipboard
| Challenge: | Document-level event extraction aims to extract structured information from unstructured text. |
| Approach: | They propose a cross-document event extraction pipeline that integrates event information from multiple documents and provides a comprehensive perspective on events. |
| Outcome: | The proposed pipeline achieves about 72% F1 in end-to-end cross-document event extraction, setting up a benchmark for future research. |
The Connection between the Text and Images of News Articles: New Insights for Multimedia Analysis (2020.lrec-1)
Copied to clipboard
| Challenge: | a case study of text and images reveals the inadequacy of simplistic assumptions about their connection and interplay. |
| Approach: | They propose to use a case study to analyze 1000 flood-related news articles . they find that articles cluster into seven categories related to different topical aspects of flooding . |
| Outcome: | The results show that flood-related news articles do not consistently report on a single, currently unfolding flooding event. |
Joint Detection and Coreference Resolution of Entities and Events with Document-level Context Aggregation (2021.acl-srw)
Copied to clipboard
| Challenge: | Recent work on extracting information from sentences or paragraphs has a difficulty analyzing longer contexts. |
| Approach: | They propose a jointly trained model that can be used for various information extraction tasks at the document level. |
| Outcome: | The proposed model improves entity and event typing and typing on documents from the ACE05-E+ dataset. |
Text-to-Text Extraction and Verbalization of Biomedical Event Graphs (2022.coling-1)
Copied to clipboard
| Challenge: | Biomedical events represent complex, graphical, and semantically rich interactions expressed in the scientific literature. |
| Approach: | They propose a framework to solve event extraction and event verbalization with a unified text-to-text approach. |
| Outcome: | The proposed framework achieves greater state-of-the-art performance than single-task competitors and can generate coherent natural language utterances from structured data. |
Automatic Data Acquisition for Event Coreference Resolution (2021.eacl-main)
Copied to clipboard
| Challenge: | lexical paraphrases and high precision rules informed by news discourse structure can be used to collect coreferential and non-coreferential event pairs from unlabeled English news articles. |
| Approach: | They propose to use lexical paraphrases and news discourse structure to automatically collect coreferential and non-coreferential event pairs from unlabeled English news articles. |
| Outcome: | The proposed model performs better than the supervised model on evaluation datasets with different event domains and text genres. |
A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to summarize video content have only considered video and image data, and the trend towards multimodal video summarization is changing. |
| Approach: | They propose a multimodal video summarization task setting and a dataset to train and evaluate the task. |
| Outcome: | The proposed task is useful as a practical application and presents a highly challenging problem worthy of study. |
Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles (2024.lrec-main)
Copied to clipboard
Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Arthur Baker, Rahul Ghosh, James H. Martin, Nathaniel Blanchard, Nikhil Krishnaswamy
| Challenge: | Existing methods for cross-document coreference resolution do not provide images for all mentions of events. |
| Approach: | They propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. |
| Outcome: | The proposed method improves on a popular ECB+ and AIDA datasets. |
Multi-Document Event Extraction Using Large and Small Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multi-document event extraction have limited attention . despite its practical significance, this task has inherent challenges . |
| Approach: | They propose a collaborative framework that integrates large language models for multi-step reasoning and fine-tuned small language models to handle key subtasks. |
| Outcome: | The proposed framework outperforms existing methods and provides new insights into collaborative reasoning to tackle the complexities of multi-document event extraction. |