Papers by Ryan He
Let’s Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023.emnlp-main)
Copied to clipboard
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Wang
| Challenge: | Existing studies show vision-language systems can reason about images using natural language, but their capacity for video reasoning remains underexplored. |
| Approach: | They propose to frame video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language systems' capacity to reason about images using natural language. |
| Outcome: | The proposed models can generate multiple intermediate keyframes and predict future keyframe, and they perform poorly on GPT-4, GPT-3, and VICUNA. |
When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing static benchmarks that measure task performance often rely on a simple input-output configuration. |
| Approach: | They propose an evaluation pipeline that evaluates code models with different feedback types in an interactive setting. |
| Outcome: | The proposed evaluation pipeline compares model-user collaboration with static benchmarks by obfuscating inputs to a simulated user. |
Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve language models' performance do not exploit this phenomenon . |
| Approach: | They propose to use contextual information to select and construct prompts that improve model performance. |
| Outcome: | The proposed methods show that the mutual information between a context and a question is an effective gauge for language model performance. |