Papers by Kate Saenko
Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to defend against fake news are limited to text and metadata . authors identify weaknesses that adversaries can exploit by manipulating such technology . |
| Approach: | They propose a more realistic defense mechanism to defend against machine-generated news . they use a NeuralNews dataset to identify weaknesses that adversaries can exploit . |
| Outcome: | The proposed approach detects visual-semantic inconsistencies and provides a useful first line of defense against machine-generated disinformation. |
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing models that ground language into visual appearance and route structure are outperforming their visual counterparts in unseen new environments. |
| Approach: | They propose to decompose the grounding procedure into a set of expert models with access to different modalities and ensemble them at prediction time. |
| Outcome: | The proposed model outperforms models with only route structure and visual features on the benchmark Room-to-Room dataset. |
Tell Me What’s Next: Textual Foresight for Generic UI Representations (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities. |
| Approach: | They propose a pretraining objective for learning UI screen representations using captioning. |
| Outcome: | The proposed approach outperforms state-of-the-art on generation tasks with 28x fewer images. |
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (2023.emnlp-main)
Copied to clipboard
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
| Challenge: | Existing datasets for webpages contain only fragments of webpages . generative tasks like page description generation and section summarization are often left unstudied . |
| Approach: | They introduce a Wikipedia Webpage suite that contains 2M pages with all associated image, text, and structure data. |
| Outcome: | The proposed approach performs better than full attention with lower computational complexity. |
Object Hallucination in Image Captioning (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning metrics do not capture image relevance . current metrics only measure similarity to ground truth captions . |
| Approach: | They propose a new image relevance metric to evaluate captioning models with veridical visual labels and assess their rate of object hallucination. |
| Outcome: | The proposed metrics show that models with veridical visual labels have higher hallucination rates than models with lower hallucinosity. |
Scaling Up Temporal Domain Generalization via Temporal Experts Averaging (2025.emnlp-main)
Copied to clipboard
Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer
| Challenge: | Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. |
| Approach: | They propose a framework that updates the entire model using weight averaging to maximize generalization potential while minimizing computational costs. |
| Outcome: | The proposed framework outperforms previous methods by up to 69% while being up to 60x more efficient. |