Papers by Grace Luo
Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation (2022.naacl-main)
Copied to clipboard
| Challenge: | Detecting out-of-context media is a problem in domains of public significance . a method that leverages automatically generated hard image-text mismatches is proposed . |
| Approach: | They propose a method that leverages automatically generated hard image-text mismatches to detect out-of-context media . they analyze tweets relevant to topics such as COVID-19, Climate Change and Military Vehicles . |
| Outcome: | The proposed method improves detection accuracy over a strong baseline on a set of fakes created by humans. |
G3: Geolocation via Guidebook Grounding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new task uses explicit knowledge from human-written guidebooks to improve geolocation accuracy . a state-of-the-art image-only method is unable to predict the location of an image . |
| Approach: | They propose a task that uses streetview images and a guidebook to predict a country for each image . they add clues from the guidebook and supervise attention with country-level pseudo labels . |
| Outcome: | The proposed method outperforms state-of-the-art image-only geolocation methods with 5% improvement in Top-1 accuracy. |
Focus! Relevant and Sufficient Context Selection for News Image Captioning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work only coarsely leverages the article to extract the necessary context, which makes it difficult for models to identify relevant events and named entities. |
| Approach: | They propose to use a vision and language retrieval model CLIP to localize the visually grounded entities in the news article and then capture the non-visual entities via an open relation extraction model. |
| Outcome: | The proposed model significantly improves on existing models and achieves state-of-the-art on multiple benchmarks. |
NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media (2021.emnlp-main)
Copied to clipboard
| Challenge: | a threat scenario where an image is used out of context to support a narrative is proposed. |
| Approach: | They propose a dataset where both image and text are unmanipulated but mismatched . they benchmark several state-of-the-art multimodal models on their dataset . |
| Outcome: | The proposed dataset shows that machine-driven image repurposing is now a realistic threat . it provides samples that represent challenging instances of mismatch between text and image . |