Papers by Alex Mei
Let’s Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023.emnlp-main)
Copied to clipboard
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Wang
| Challenge: | Existing studies show vision-language systems can reason about images using natural language, but their capacity for video reasoning remains underexplored. |
| Approach: | They propose to frame video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language systems' capacity to reason about images using natural language. |
| Outcome: | The proposed models can generate multiple intermediate keyframes and predict future keyframe, and they perform poorly on GPT-4, GPT-3, and VICUNA. |
ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not provide robustness evaluations for large language models, but we find that they are inconsistent in performance. |
| Approach: | They propose to use semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection to generate a test suite of prompts covering diverse robustness settings. |
| Outcome: | The proposed system generates a set of prompts covering diverse settings covering semantic equivalence, related scenarios, and adversarial. |
Learning to Prioritize: Precision-Driven Sentence Filtering for Long Text Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Neural text summarization models are limited by their maximum input length, posing a challenge to summarizing longer texts comprehensively. |
| Approach: | They propose a pre-processing layer that removes low-quality sentences in articles to improve existing summarization models. |
| Outcome: | The proposed approach improves state-of-the-art summarization models on WikiHow and Reddit TIFU datasets by 3.84 and 8.57 points on the full test set and the long article subset. |
Mitigating Covertly Unsafe Text within Natural Language Systems (2022.findings-emnlp)
Copied to clipboard
Alex Mei, Anisha Kabir, Sharon Levy, Melanie Subbiah, Emily Allaway, John Judge, Desmond Patton, Bruce Bimber, Kathleen McKeown, William Yang Wang
| Challenge: | Existing studies on text safety have focused on overtly unsafe, covertly, or indirectly unsafe statements. |
| Approach: | They propose a method to identify physical harm-causing statements as overtly, covertly or indirectly unsafe and a solution to mitigate the generation of such statements. |
| Outcome: | The proposed methods identify the type of unsafe language that can cause physical harm and identify mitigation strategies to inspire future researchers to tackle this challenging problem. |
Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI (2023.findings-acl)
Copied to clipboard
| Challenge: | Covertly unsafe text is an area of particular interest as it is difficult to detect as harmful . previous work focused on explicit violent text and typically expressed through violent keywords. |
| Approach: | They propose a framework that leverages external knowledge for trustworthy rationale generation in the context of safety. |
| Outcome: | The proposed framework improves safety classification accuracy by 5.9% on the SafeText dataset, and shows that it is more accurate than previous frameworks. |